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FROM  THE 
CHAIRMAN  AND 
EXECUTIVE 
EDITOR 

Dr.  Larrie  D.  Ferreiro 


The  theme  for  this  edition  of  Defense 
Acquisition  Research  Journal  is  “Learning 
from  the  Past.”  As  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and 
Logistics  Frank  Kendall  noted  in  2011, 
the  Better  Buying  Power  initiatives  were 
not  so  much  a  collection  of  novel  ideas  as 
they  were  guidelines  “distilled  from  best 
practices  and  lessons  learned.”1  He  also 
reminded  the  acquisition  workforce  in 
his  rollout  of  the  revised  DoD  Instruction 
5000.02  in  January  2015,  that  "we  will  never  stop  learning  from 
our  experience."2 

This  issue  begins  with  a  rarely  seen  feature  in  the  pages  of  Defense 
ARJ—  Letters  to  the  Editor.  We  often  receive  comments  about  the 
articles  published  in  this  Journal,  but  rarely  are  we  afforded  the 
opportunity  to  publish  them.  In  this  issue  we  present  a  reader’s 
comments,  and  the  authors’  subsequent  reply,  to  an  article  from  the 
July  2013  issue,  “Current  Barriers  to  Successful  Implementation 
of  FIST  Principles.”  We  appreciate  and  encourage  this  level  of 
open  discourse  on  topics  of  immediate  interest  to  the  Defense 
Acquisition  Workforce. 


July  2015 


In  keeping  with  the  theme  of  learning  from  the  past,  the  first  two 
articles  are  reprints  from  previous  issues,  but  which  continue  to 
have  relevance  today.  In  “Cost  Overrun  Optimism:  Fact  or  Fiction?” 
by  Maj  David  D.  Christensen,  USAF,  (originally  published  in  1994), 
and  Leland  G.  Jordan's  “Systemic  Fiscal  Optimism  in  Defense 
Planning”  (published  in  2000),  the  authors  identified  systematic 
underestimating  of  cost  growth  and  systematic  overestimating  of 
resource  availability  as  major  contributing  factors  to  inaccurate  and 
unrealistic  cost  estimates.  This  dilemma  is  not  limited  to  defense 
programs,  but  exists  in  any  complex  system  acquisition;  in  the  book 
Megaprojects  and  Risk:  An  Anatomy  of  Ambition  by  Brent  Flyvbjerg 
et  al.,  reviewed  in  Defense  ARJ  (Issue  No.  59,  July  2011,  p.  336), 
which  examines  three  large  European  civil  engineering  programs, 
the  authors  cite  “overoptimistic  estimates”  as  being  primary  causes 
for  cost  and  schedule  overruns. 

Col  Dennis  J.  Rensel,  USAF  (Ret.),  in  “Resilience— A  Concept,”  takes 
a  holistic  approach  to  measuring  the  “health”  of  systems  and  capa¬ 
bilities.  In  “Performance  Indexing:  Assessing  the  Nonmonetized 
Returns  on  Investment  in  Military  Equipment,”  the  authors  Ian  D. 
MacLeod  and  Capt  Robert  A.  Dinwoodie,  USMC,  tackle  the  problem 
of  calculating  the  “worth”  of  investments  in  military  equipment  pro¬ 
grams  when  a  direct  comparison  using  monetary  returns  falls  short. 

The  featured  book  in  this  issue’s  Defense  Acquisition  Professional 
Reading  List  is  Richard  Whittle’s  The  Dream  Machine:  The 
Untold  History  of  the  Notorious  V-22  Osprey,  reviewed  by  Defense 
Acquisition  University  Professor  Owen  Gadeken. 


1  Kendall,  F.  (2011,  September-October).  Better  buying  power:  Foreword. 
Defense  AT&L  Magazine,  40(5),  2-4. 

2  Kendall,  F.  (2015,  January  7).  Department  of  Defense  Instruction 
5000.02  [Memorandum],  Washington,  DC:  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  &  Logistics. 
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LETTERS  TO 
THE  EDITOR 


From  the  Executive  Editor,  Defense  ARJ 

We  often  receive  comments  about  the  articles  published  in  the 
Defense  ARJ,  but  rarely  have  we  been  afforded  the  opportunity  to 
publish  them.  In  July  2013  we  published  the  article  “Current  Barriers 
to  Successful  Implementation  of  FIST  Principles”  by  Brandon  Keller 
and  J.  Robert  Wirthlin  (abstract  and  link  below).  We  received  cri¬ 
tiques  from  Dan  Ward,  one  of  the  sources  that  the  authors  cite  in 
their  article.  After  contacting  the  authors,  we  offered  to  publish  Lt 
Col  Ward’s  critique  along  with  a  response  from  the  original  authors. 
Both  parties  agreed,  and  their  letters  are  presented  here. 

Current  Barriers  to  Successful  Implementation  of 
FIST  Principles 

Capt  Brandon  Keller,  USAF,  and  Lt  Col  J.  Robert  Wirthlin,  USAF 
Defense  ARJ,  July  2013,  Vol.  20  No.  2: 194-217 
http://www.dau.mil/publications/DefenseARJ/ARJ/ARJ66/ 
ARJ_66-Keller.pdf 

Abstract:  The  Fast,  Inexpensive,  Simple,  and  Tiny  (FIST)  frame¬ 
work  proposes  a  broad  set  of  organizational  values,  but  provides 
limited  guidance  on  practical  implementation.  Implementing  FIST 
principles  requires  clarifying  the  definitions  of  “fast,”  “inexpen¬ 
sive,”  and  “simple,”  recognizing  where  FIST  does  and  does  not 
apply.  Additionally,  a  subset  of  the  FIST  heuristics  was  expanded 
upon  to  increase  their  usefulness  for  practitioners.  The  primary 
research  findings  are  that  FIST  principles  are  less  conducive  for 
highly  complex  or  novel  systems,  immature  technologies,  future 
needs,  acquisitions  in  early  development  phases,  or  when  perfor¬ 
mance  is  the  foremost  value.  FIST  principles  were  also  found  to  be 
constrained  by  the  acquisition  process,  the  requirements  process, 
and  oversight. 
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Letter  from  Lt  Col  Dan  Ward,  USAF 

To  The  Editor, 

There  are  three  significant  misrepresentations  in  the  2013  arti¬ 
cle  “Current  Barriers  to  Successful  Implementation  of  FIST 
Principles,”  by  Keller  and  Wirthlin.  I  would  like  to  offer  the  follow¬ 
ing  corrections: 

1.  The  authors  misrepresent  my  opinion  several  times,  saying 
“Ward  agrees”  to  propositions  I  disagree  with.  Specifically,  the 
authors  claim  I  believe  FIST  is  “not  conducive  for  immature 
technologies”  and  that  using  mature  technologies  “is  often  the 
antithesis  of  innovation.”  This  demonstrates  a  shallow  reading 
of  the  FIST  literature  and  a  misunderstanding  of  the  nature  of 
innovation  itself.  Many  writers,  including  myself,  explain  that 
innovation  often  results  from  putting  mature  technologies 
together  in  new  and  interesting  ways.  Using  mature  components 
is  therefore  entirely  consistent  with  delivering  innovative  new 
capabilities  to  the  marketplace. 

2.  The  authors  misrepresent  the  scope  of  their  own  research.  They 
claim  to  have  reviewed  “the  multitude  of  materials  related  to 
FIST,”  but  the  most  recent  document  they  cited  was  published 
in  2009.  They  therefore  omitted  four  years  of  publications  on  the 
topic,  upwards  of  25  articles,  journal  papers,  conference  presen¬ 
tations,  and  other  material  published  by  myself  and  others.  Such 
shallow  research  presents  an  incomplete  picture  of  the  topic 
and  does  a  disservice  to  readers.  It  also  helps  explain  why  their 
portrayal  of  FIST  was  so  far  off  the  mark  and  why  they  claimed 
“no  evidence  is  offered”  and  that  FIST  “provides  limited  guid¬ 
ance  on  practical  implementation.”  A  more  complete  survey  of 
the  recent  literature  would  have  revealed  considerable  evidence 
and  guidance. 

3.  Finally,  the  authors  misrepresent  their  contribution  to  the  FIST 
concept.  They  claim  to  have  “expanded  upon”  the  FIST  heuris¬ 
tics  and  offered  some  “recommended  additions,”  but  fully  half 
of  their  additions  either  cite  my  work,  lightly  paraphrase  my 
work  without  citation,  or  are  copied  word-for-word  from  my 
work  without  citation.  For  example,  one  of  the  heuristics  they 
claimed  to  add  was  the  following:  “The  project  leader’s  influence 
over  the  development  is  inversely  proportional  to  the  budget  and 
schedule.”  That  exact  line  appears  on  page  103  of  my  thesis,  but 
the  authors  present  it  as  their  own  original  contribution. 
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The  article  is  riddled  with  other  errors,  largely  resulting  from  shal¬ 
low  research  (see  #2  above).  It  attributes  the  concept  of  “disruptive 
innovation”  to  a  2011  publication  by  Dyer  instead  of  a  1995  paper 
by  Christensen.  It  asserts  “early  operator  feedback  on  a  satellite 
program...  is  nearly  impossible,”  which  overlooks  the  successful 
Operationally  Responsive  Space  office,  established  in  2007.  It  claims 
“FIST  is  less  conducive  for  complex,  large  programs,”  which  over¬ 
looks  the  FIST  Navy’s  Virginia  Class  submarine  program. 

While  I  enthusiastically  welcome  discussions  and  debates  about 
FIST,  I  am  disappointed  in  the  way  Keller  and  Wirthlin  misrepre¬ 
sented  my  opinions,  my  work,  and  their  own  contribution  to  the  topic. 
They  incorrectly  attributed  opinions  to  me  which  I  do  not  share, 
overlooked  four  years’  worth  of  material,  and  claimed  my  words 
as  their  own.  I  hope  future  writers  do  not  follow  their  example  and 
instead  present  a  more  accurate,  thorough,  and  original  contribution. 

Response  from  Capt  Brandon  Keller,  USAF,  and  Lt  Col  J. 
Robert  Wirthlin,  USAF 

To  the  Editor, 

We  appreciate  Defense  ARJ giving  us  the  opportunity  to  respond  to 
Lt  Col  Ward’s  Letter  to  the  Editor  about  our  article  in  the  July  2013 
Defense  ARJ  edition.  We  have  a  professional  disagreement  with 
aspects  of  FIST  that  we  documented  in  this  peer-reviewed  forum 
and  still  assert  to  be  true.  Our  response  to  his  remarks  follows: 

1.  We  never  wrote  that  Ward  agrees  that  FIST  is  not  condu¬ 
cive  for  immature  technologies...  we  say  that  Ward  agrees 
“that  a  key  to  FIST  implementation  is  the  use  of  mature 
technologies”  (Ward,  2009).  The  finding  “FIST  is  not  con¬ 
ducive  for  immature  technologies”  is  our  conclusion; 
however,  although  Ward  says  he  disagrees  with  the  state¬ 
ment,  pp.  16,  17,  32,  44,  89,  and  90  of  Ward’s  thesis  from 
2009  clearly  promote  the  use  of  mature  technology  in  FIST. 

We  also  agree  combining  mature  technology  in  novel  ways  can 
produce  innovative  results. 

2.  We  did  not  misrepresent  anything.  The  most  recent  cited  docu¬ 
ment  in  our  published  work  was  from  2012,  but  there  are  several 
reasons  the  most  recent  FIST-related  citation  was  2009: 
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a.  We  focused  on  citing  peer-reviewed  publications  for  aca¬ 
demic  rigor;  and, 

b.  We  tried  to  cite  the  original  document  in  which  a  viewpoint 
was  presented— many  newer  publications  merely  re-stated 
previously  documented  FIST  viewpoints.  Quantity  of  cita¬ 
tions  about  FIST  does  not  strengthen  the  veracity  of  the 
assertions. 

c.  Concern  over  whether  we  did  not  attribute  conclusions  by 
other  authors  from  earlier  dates  should  not  imply  we  were 
unaware  of  those  authors’  conclusions,  nor  does  it  demon¬ 
strate  academic  malpractice.  Researchers  in  similar  fields 
often  draw  the  same  conclusions  independently. 

3.  We  did  miss  one  citation.  In  August  2013,  we  privately  apolo¬ 
gized  for  the  oversight  of  the  heuristic;  it  was  an  honest  oversight 
on  our  part  and  in  no  way  intended  to  imply  his  work  as  ours. 

There  are  clear  distinctions  between  FIST  and  our  own  conclusions, 
and  we  disagree  that  half  of  the  heuristics  come  from  Ward’s  body  of 
work.  The  heuristics  in  our  article  that  do  come  from  others’  work 
(and  cited  as  such)  are  included  because  we  believe  they  further 
solidify  our  own  positions.  This  also  includes  heuristics  cited  from 
Ward  that  add  credence  to  our  findings. 

It  is  unfortunate  that  Ward  uses  a  strawman  of  embellished  asser¬ 
tions  designed  to  discredit  our  work  without  directly  confronting 
the  conclusions  or  presenting  any  evidence  that  would  further 
enlighten  discussion  and  knowledge  of  the  subject  matter.  Rigorous 
academic  research  ought  to  question  and  test  the  assertions  of  the 
author.  Our  work  invites  other  researchers  to  join  the  discussion. 
FIST  concepts  have  been  around  a  very  long  time  in  the  project 
management  profession,  but  citing  a  handful  of  successful  programs 
doesn’t  show  causality  of  success  when  compared  to  the  thousands 
of  unsuccessful  programs.  It  is  far  more  the  norm  for  the  current 
barriers  to  FIST  implementation  to  surface  than  FIST  principles 
leading  to  a  successful  program  by  itself.  We  still  assert  that  bar¬ 
riers  exist  to  successful  implementation  of  FIST  in  all  types  of 
defense  acquisition  scenarios. 
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The  Defense  Acquisition  Research  Agenda  is  intended  to  make 
researchers  aware  of  the  topics  that  are,  or  should  be,  of  particular 
concern  to  the  broader  defense  acquisition  community  throughout 
the  government,  academic,  and  industrial  sectors.  The  purpose  of 
conducting  research  in  these  areas  is  to  provide  solid,  empirically 
based  findings  to  create  a  broad  body  of  knowledge  that  can  inform 
the  development  of  policies,  procedures,  and  processes  in  defense 
acquisition,  and  to  help  shape  the  thought  leadership  for  the  acqui¬ 
sition  community. 

Each  issue  of  the  Defense  ARJ  will  include  a  different  selection  of 
research  topics  from  the  overall  agenda,  which  is  at:  http://www. 
dau.mil/research/Pages/researchareas.aspx 

Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to  measure 
the  effect  on  defense  acquisition  costs  of  maintaining 
an  industrial  base  in  various  sectors? 

•  What  means  exist  (or  can  be  developed)  of  measuring 
the  effect  of  utilizing  defense  industrial  infrastructure 
for  commercial  manufacture  in  growth  industries?  In 
other  words,  can  we  measure  the  effect  of  using  defense 
manufacturing  to  expand  the  buyer  base? 

•  What  means  exist  (or  can  be  developed)  to  determine 
the  degree  of  openness  that  exists  in  competitive 
awards? 
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•  What  are  the  different  effects  of  the  two  best-value 
source-selection  processes  (tradeoff  vs.  lowest  price 
technically  acceptable)  on  program  cost,  schedule,  and 
performance? 

Strategic  Competition 

•  Is  there  evidence  that  competition  between  system 
portfolios  is  an  effective  means  of  controlling  price 
and  costs? 

•  Does  lack  of  competition  automatically  mean  higher 
prices?  For  example,  is  there  evidence  that  sole  source 
can  result  in  lower  overall  administrative  costs  at  both 
the  government  and  industry  levels,  to  the  effect  of 
lowering  total  costs? 

•  What  are  the  long-term  historical  trends  for  compe¬ 
tition  guidance  and  practice  in  defense  acquisition 
policies  and  practices? 

•  To  what  extent  are  contracts  being  awarded  non- 
competitively  by  congressional  mandate,  for  policy 
interest  reasons?  What  is  the  effect  on  contract  price 
and  performance? 

•  What  means  are  there  (or  can  be  developed)  to  deter¬ 
mine  the  degree  to  which  competitive  program  costs 
are  negatively  affected  by  laws  and  regulations  such  as 
the  Berry  Amendment  and  Buy  American  Act? 
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Cost  Overrun 

Optimism: 

FACT  or 

FICTION? 

1  Maj  David  D.  Christensen,  USAF 


Program  managers  are  advocates  by  necessity.  When  taken  to  the  extreme, 
program  advocacy  can  result  in  the  suppression  of  adverse  information  about 
the  status  of  a  program.  Such  was  the  case  in  the  Navy’s  A-12  program.  In 
“A-12  Administrative  Inquiry”  Beach  (1990)  speculates  that  such  “abiding 
cultural  problems”  were  not  unique  to  the  Navy.  To  test  that  assertion,  this 
article  examines  cost  overrun  data  on  64  completed  acquisition  contracts 
extracted  from  the  Defense  Acquisition  Executive  Summary  database.  Cost 
overruns  at  various  contract  completion  points  are  compared  with  projected 
final  cost  overruns  estimated  by  contractor  and  government  personnel. 
The  comparison  shows  that  the  overruns  projected  by  the  contractor  and 
government  were  excessively  optimistic  throughout  the  lives  of  the  contracts 
examined.  These  results  were  found  insensitive  to  contract  type  (cost, 
price),  contract  phase  (development,  production),  the  type  of  weapon 
system  (air,  ground,  sea),  and  the  military  service  (Air  Force, 

Army,  Navy)  that  managed  the  contract. 


Keywords:  comparison,  management,  performance,  projected, 
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Cost  Overrun  Optimism: 

Fact  or  Fiction? 

According  to  Gansler  (1989,  p.  4),  the  average  cost  overrun  on  a  major 
defense  contract  has  been  about  40  percent.  Although  some  of  the  causes 
of  cost  overruns  are  beyond  the  control  of  program  managers,  supporting 
an  unrealistically  low  estimate  of  the  final  cost  of  a  defense  contract  can 
only  harm  the  program  in  the  long  run.  The  cancellation  of  the  Navy’s  A-12 
program  in  January  1991  is  a  highly  publicized  example  of  this  problem. 

Chester  P.  Beach  (1990),  the  Inquiry  Officer  of  the  A-12  cancellation, 
reported  that  pessimistic  projections  regarding  the  program’s  cost  were 
suppressed  to  protect  the  program  and  the  careers  of  key  managers.  When 
Secretary  of  Defense  Dick  Cheney  canceled  the  program  in  January  1991,  he 
complained  that  no  one  could  tell  him  its  final  cost  (Morrison,  1991).  In  fact, 
there  were  many  estimates  of  the  program’s  completion  cost:  some  estimates 
were  more  than  $1  billion  higher  than  the  ones  supported  by  the  govern¬ 
ment  program  office  and  by  the  contractors.  The  problem  was  the  delayed 
and  reluctant  communication  of  the  pessimistic  estimates  to  key  decision 
makers  above  the  government  program  office.  Although  no  one  can  say  with 
certainty  that  the  timely  communication  of  more  realistic  estimates  would 
have  saved  the  A-12,  it  seems  likely  that  at  least  part  of  the  $1.35  billion  in 
excess  progress  payments  made  to  the  contractors  could  have  been  avoided 
(Ferber  &  Math,  1991). 

More  realistic  estimates  and  a  culture  that  will  tolerate  them  are  needed. 
Program  managers/directors  are  necessarily  advocates  of  their  programs. 
However,  program  advocacy  is  no  excuse  for  suppressing  critical  infor¬ 
mation  about  a  program’s  cost,  schedule,  or  technical  performance.  In  an 
acquisition  policy  letter,  J.  J.  Welch  (1991),  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  wrote: 

A  program  director  (PD)  must  be  an  advocate  of  his  or  her 
program....The  PD’s  advocacy  must  not  cross  the  line  into 
attempting  to  “sell”  the  program,  but  must  clearly  be  viewed 
as  supportive  to  the  user’s  requirements.  The  PD  must  artic¬ 
ulate  the  pros  and  cons,  as  well  as  the  “maturity  curve” 
status,  in  a  clear  and  comprehensive  manner  to  preclude 
unfulfilled  expectations  or  surprises.  Such  advocacy  must 
be  based  on  honesty  and  integrity  to  accurately  portray 
program  status.1 
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Regardless  of  this  policy  statement,  Gansler  (1989,  p.  212)  reports  that  the 
majority  of  program  managers’  time  is  spent  “selling”  their  programs  to 
budget  committees.  In  addition,  research  has  shown  that,  once  a  program 
is  more  than  15  percent  to  20  percent  complete,  it  is  highly  unlikely  that 
the  final  cost  overrun  will  be  less  than  the  present  cost  overrun  (W.  Abba, 
personal  communication,  1992;  Christensen  &  Payne,  1992;  Heise,  1991; 
Wilson,  1991).  Despite  these  facts,  contractor  and  government  program 
managers  often  claim  optimistically  that  dramatic  recoveries  from  cost 
overruns  are  possible. 


Using  information  extracted  from  the  Defense  Acquisition 
Executive  Summary  (DAES)  database,  this  article  docu¬ 
ments  the  optimistic  forecasts  of  contract  completion 
costs  on  64  completed  contracts.  Average  cost  overruns 
at  various  contract  completion  points  are  compared 
with  projected  final  cost  overruns  estimated  by  con¬ 
tractor  and  government  personnel.  The  comparison 
shows  that  the  overruns  projected  by  the  contractor  and 
government  were  exceedingly  optimistic  throughout  the 
lives  of  the  contracts  examined.  These  results  were  found 
insensitive  to  contract  type  (cost,  price),  contract  phase  (devel¬ 
opment,  production),  the  type  of  weapon  system  (air,  ground, 
sea),  or  the  military  service  that  managed  the  contract. 


Background 


Cost  overruns  and  projected  final  overruns  are  regularly 
reported  on  cost  management  reports  prepared  by  the  contractor. 

These  reports  include  the  Cost  Performance  Report  (CPR)  and 
the  Cost/Schedule  Status  Report  (C/SSR).  Department  of  Defense 
Instruction  5000.2  stipulates  that  a  CPR  be  submitted  for  contracts 
that  require  compliance  with  the  Department  of  Defense  (DoD)  cost/ 
schedule  control  systems  criteria  (C/SCSC)  (Department  of  Defense, 
1991).  For  contracts  not  required  to  comply  with  the  criteria,  the  C/SSR 
is  usually  required.2 


Cost/schedule  control  systems  criteria  are  not  a  management  system. 
Instead,  they  establish  minimal  standards  for  the  management  control 
systems  used  by  the  contractor  and  have  two  objectives: 
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1.  For  contractors  to  use  effective  internal  cost  and  schedule  management 
control  systems;  and 

2.  For  the  government  to  be  able  to  rely  on  timely  and  auditable  data 
produced  by  those  systems  for  determining  product-oriented  contract 
status  (Department  of  the  Air  Force,  1989). 

Implicit  in  these  objectives  is  the  assumption  that,  if  the  contractor’s  man¬ 
agement  control  systems  comply  with  the  criteria,  the  data  generated  by 
those  systems  are  reliable  (Christensen,  1989). 

Data  summarizing  a  contract’s  cost  and  schedule  performance  are  listed  in 
the  cost-management  report.  Key  data  elements  of  the  report  are  shown  in 
Figure  1.  The  budgeted  cost  of  work  scheduled  (BCWS)  is  the  sum  of  bud¬ 
gets  allocated  to  time -phased  elements  of  work  on  the  contract,  known  as 
work  packages  and  planning  packages.  The  cumulative  expression  of  these 
budgets,  the  performance  measurement  baseline,  takes  on  a  characteristic 
S-shaped  curve.  The  end  point  of  the  baseline,  the  budget  at  completion 
(BAC),  represents  the  total  budget  of  all  the  identified  work  on  the  contract. 

As  shown  in  the  figure,  the  contractor  also  reports  an  estimate  of  the  final 
cost  of  the  contract,  termed  the  estimate  at  completion  (EAC).  The  EAC  is 
an  extrapolation  of  the  cumulative  actual  cost  of  work  performed  (ACWP) 
to  the  end  of  the  contract.  If  the  projected  final  cost  differs  from  the  total 
budget,  the  contractor  is  predicting  a  cost  overrun  at  completion.  It  is  often 
revealing  to  compare  the  predicted  cost  overrun  at  completion  to  the  present 
cost  overrun.  If  the  present  overrun  is  worse  than  the  predicted  final  over¬ 
run,  the  contractor  is  predicting  effectively  that  the  cost  of  the  remaining 
work  on  the  contract  will  be  less  than  budgeted.  For  this  article,  the  present 
cost  overrun  is  defined  as  the  difference  between  the  cumulative  budgeted 
cost  for  work  performed  (BCWP)  and  the  cumulative  ACWP  (see  Figure 
1).  The  BCWP  is  the  same  number  as  BCWS,  but  is  recorded  when  work  is 
actually  accomplished.  Clearly,  if  the  cost  of  the  completed  work  exceeds 
the  budget,  a  cost  overrun  is  identified.  If  the  cost  overrun  is  significant,  it 
is  investigated  to  determine  the  cause.  Hopefully,  the  timely  and  disciplined 
analysis  of  significant  overruns  will  result  in  corrective  action  before  the 
problems  become  serious. 

The  effectiveness  of  variance  analysis  depends  on  organizational  culture. 
In  a  healthy  culture  a  variance  is  considered  an  opportunity  for  improve¬ 
ment.  In  an  unhealthy  culture  a  variance  is  bad  news,  and  individuals  or 
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even  organizations  responsible  for  unfavorable  variances  may  be  punished. 
The  result  of  this  “shoot  the  messenger”  culture  can  be  the  suppression  of 
adverse  information  about  a  contract's  status. 

Although  routine  analysis  in  the  A-12  program  revealed  adverse  trends,  the 
significance  of  the  unfavorable  cost  and  schedule  variances  was  not  revealed 
to  senior  civilian  decision  makers  above  the  government  program  office. 
According  to  Beach  (1990),  the  projected  final  completion  costs  supported  by 
the  contractor  and  the  government  program  manager  were  unrealistic.  For 
example,  at  the  37  percent  completion  point,  the  A-12  contractors  reported 
a  cost  overrun  of  $459  million  and  a  projected  cost  overrun  at  completion 
of  $354  million  (Campbell  &  Fleming,  1991). 


FIGURE  1.  ADVERSE  COST  VARIANCE,  TERMED  COST  OVERRUN 


The  government  program  manager's  estimated  final  overrun  was  slightly 
higher  than  the  contractor  estimate  yet  less  than  the  overrun  to  date. 

Apparently  the  need  to  present  an  optimistic  picture  was  a  dominant  con¬ 
sideration  that  effectively  suppressed  more  realistic  estimates.  Near  the  end 
of  his  report,  Beach  (1990)  speculates  that  this  “abiding  cultural  problem” 
was  not  specific  to  the  A-12,  but  was  a  problem  common  to  other  major 
defense  programs: 
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There  is  no  reason  to  believe  that  the  factors  which  made 
these  officials  respond  the  way  they  did  was  unique  to  this 
military  department.  Indeed,  experience  suggests  that  they 
are  not.  Unless  means  can  be  found  to  solve  this  abiding 
cultural  problem,  the  failures  evidenced  in  this  report  can 
be  anticipated  to  occur  again  in  the  same  or  a  similar  man¬ 
ner.  (p.  27) 

This  article  provides  evidence  that  supports  this  assertion  by  examining 

available  cost  data  on  completed  contracts. 


Methodology 

The  purpose  of  this  study  was  to  determine  if  the  overruns  at  comple¬ 
tion  projected  by  contractor  and  government  personnel  are  unrealistically 
optimistic.  Research  has  established  that,  once  a  contract  is  15  percent  com¬ 
plete,  the  final  cost  overrun  will  exceed  the  cost  overrun  to  date  (W.  Abba, 
personal  communication,  1992;  Christensen,  1989;  Heise,  1991;  Wilson, 
1991).  Thus,  a  projected  overrun  at  completion  is  defined  as  unrealistically 
optimistic  if  it  is  less  than  the  present  cost  overrun. 

To  test  the  hypothesis,  averages  of  the  present  cost  overrun,  the  projected 
cost  overrun  at  completion,  and  the  final  cost  overrun  were  computed  from 
a  sample  of  64  completed  contracts  extracted  from  the  DAES  database 
(Department  of  Defense,  1991).  This  database  contains  contractor  cost  and 
schedule  performance  data  on  more  than  500  defense  contracts  summa¬ 
rized  quarterly  by  government  program  offices  since  1970  (Christie,  1981). 
Because  most  of  the  contracts  in  this  database  are  C/SCSC-compliant,  the 
data  are  considered  reliable. 

Although  the  sampling  technique  was  purely  judgmental,  the  number 
and  variety  of  contracts  are  considered  sufficiently  large  to  be  general  in 
nature.  The  period  of  performance  for  these  contracts  ranged  from  1971  to 
1991.  Table  1  lists  descriptive  statistics  on  the  average  final  cost  overruns 
in  the  sample.  For  sensitivity  analysis,  the  sample  was  divided  into  several 
categories,  including  contract  type  (price,  cost),  contract  phase  (develop¬ 
ment,  production),  the  type  of  weapon  system  (air,  ground,  sea),  and  the 
Service  managing  the  contract.  For  each  category  in  the  table,  the  number 
of  contracts  and  the  average,  maximum,  and  minimum  values  for  the  final 
overrun  are  listed. 
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TABLE  1.  FINAL  COST  OVERRUN  ON  64  CONTRACTS 


Percent  OF  BUDGET  $  MILLIONS 


Category 

Number 

Avg 

Min 

Max 

Avg 

Min 

Max 

Fixed  Price 

41 

20 

-3 

109 

34 

-3 

407 

Cost 

23 

14 

-1 

46 

41 

-2 

493 

Development 

25 

21 

-1 

109 

38 

-2 

407 

Production 

39 

16 

-3 

46 

35 

-3 

493 

Air 

43 

18 

-3 

109 

45 

-3 

492 

Ground 

13 

21 

5 

45 

23 

7 

42 

Sea 

8 

12 

0 

38 

12 

0 

36 

Air  Force 

18 

19 

-1 

109 

49 

-2 

407 

Army 

28 

20 

-3 

46 

21 

-3 

46 

Navy 

18 

13 

0 

46 

47 

0 

493 

ALL 

64 

18 

-3 

109 

36 

-3 

493 

Equations  1,  2,  and  3  define  the  current  cost  overrun,  the  projected  cost 
overrun  at  completion,  and  final  cost  overrun.  Of  the  three  overruns,  only 
the  projected  cost  overrun  at  completion  is  an  estimate,  showing  the  dif¬ 
ference  between  the  budget  and  the  estimated  completion  cost.  The  others 
are  simply  the  difference  between  the  budget  and  actual  cost  of  the  work. 

Current  overrun  (CO)  =  Cumulative  (Cum)  BCWP  -  CumACWP  (1) 
Overrun  at  completion  (OAC)  =  Contract  budget  base  (CBB)  -  EAC  (2) 
Final  overrun  (FO)  =  CBB  -  Final  ACWP  (3) 

To  normalize  the  data,  the  overruns  were  converted  into  percentages  using 
Equations  4, 5,  and  6.  For  the  current  cost  overrun  percentage,  the  cumula¬ 
tive  BCWP  was  used.  For  the  others,  the  CBB  was  used.  The  CBB  is  defined 
as  the  budget  for  all  authorized  work  on  a  contract  and  includes  the  manage¬ 
ment  reserve  budget. 

Current  overrun  percentage  =  lOO*(CO/Cum  BCWP)  (4) 

Overrun  at  completion  percentage  =  100  *(OAC/CBB)  (5) 

Final  overrun  percentage  =  lOO*(FO/CBB)  (6) 
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Each  type  of  overrun  (current,  at  completion,  and  final)  was  averaged  for 
each  category  by  dividing  the  number  of  contracts  in  that  category  into  the 
total  overrun  for  that  category.  The  averaging  was  done  at  various  stages  of 
completion  ranging  from  10  to  100  percent  completed  (Equation  7). 

Percentage  completed  =  100  *(Cum  BCWP/CBB)  (7) 

Data  earlier  than  the  10  percent  completion  point  were  not  considered  suf¬ 
ficiently  reliable.  It  can  take  as  long  as  1  year  from  contract  award  for  the 
contractor  to  demonstrate  C/SCSC  compliance.  Until  then,  the  data  on  the 
cost  performance  report  are  suspect. 

As  shown  in  Table  2  in  null  form,  there  were  three  hypotheses.  Hypotheses 
HI  and  H2  compare  the  average  current  overrun  to  the  average  overrun 
at  completion  by  the  contractor  and  government  during  various  stages  of 
contract  completion.  In  hypothesis  H3,  the  average  overruns  at  completion 
by  the  contractor  and  government  are  compared. 


TABLE  2.  HYPOTHESES  TESTED 

Null  Hypothesis 

Interpretation 

HV 

CO  <  KOAC 

Contractor's  OAC  not  optimistic 

H20: 

CO  <  GOAC 

Government's  OAC  not  optimistic 

H30: 

GOAC  <  KOAC 

Government  more  optimistic  than  contractor 

Note.  KOAC  =  Contractor's  overrun  at  completion;  GOAC  =  Government's  overrun  at 
completion. 


If  hypothesis  HI  is  rejected,  the  KOAC  is  unrealistically  optimistic.  If 
hypothesis  H2  is  rejected,  the  GOAC  is  unrealistically  optimistic.  If  hypoth¬ 
esis  H3  is  rejected,  the  contractor  is  more  optimistic  than  the  government 
regarding  the  projected  overrun  at  completion.  A  one-tailed  “f  test”  was  used 
to  evaluate  each  hypothesis  at  the  95  percent  level  of  confidence. 


Results 

As  illustrated  in  Figure  2,  the  hypotheses  were  generally  confirmed. 
From  as  early  as  the  10  percent  completion  point,  the  optimism  of  the  pro¬ 
jected  cost  overrun  at  completion  is  apparent.  Throughout  the  life  of  the 
contract,  this  estimate  was  found  to  be  lower  than  the  present  and  final  cost 
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overruns.  Also  note  that  the  average  overrun  at  completion  projected  by  the 
contractor  was  more  optimistic  than  the  average  overrun  at  completion 
projected  by  the  government  program  office. 

Figure  3  shows  that  the  difference  between  the  overruns  is  statistically 
significant  through  most  stages  of  contract  completion.  When  the  one-tailed 
“t  statistic”  exceeds  a  critical  value  of  1.67  (ta  =  .05  statistic  >  1.67),  the  dif¬ 
ference  is  defined  as  significant  at  the  95  percent  level  of  confidence. 

As  illustrated  in  Figures  4  through  6,  these  results  were  generally  insensi¬ 
tive  regarding  the  contract  type,  contract  phase,  type  of  weapon  system,  and 
the  military  service  that  managed  the  contract.  To  facilitate  comparisons, 
the  scales  of  the  graphs  are  the  same.  The  statistical  significance  of  the  dif¬ 
ferences  between  the  overruns  was  generally  confirmed  for  each  category 
examined.  The  details,  however,  are  not  reported  here. 


Conclusion 

Based  on  an  analysis  of  64  completed  contracts,  the  overruns  at  comple¬ 
tion  predicted  by  the  contractor  and  by  the  government  program  office  were 
unrealistically  optimistic.  From  as  early  as  the  10  percent  completion  point 
through  the  end  of  the  contracts,  the  predicted  final  overruns  were  less  than 
the  current  overruns  reported  on  the  contracts. 


FIGURE  2.  OVERRUN  OPTIMISM  (64  CONTRACTS) 


%  Complete 
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FIGURE  3.  HYPOTHESES  CONFIRMED  (FOR  ta  =  .05  statistic  >  1.67) 


%  Complete 


Although  the  estimates  supported  by  the  government  program  offices 
were  less  optimistic  than  the  contractors’  estimates,  neither  was  found 
to  be  realistic. 

Donald  J.  Yockey  (1991),  then  Under  Secretary  of  Defense  (Acquisition), 
called  for  more  realism  throughout  the  acquisition  process,  including  esti¬ 
mating  realism. 

We  can't  afford  to  understate,  sit  on,  or  cover  up  problems  in 
any  program-at  any  time-at  any  level.  They  must  be  brought 
forward.  This  includes  not  just  ‘show  stoppers’  but  also 
‘show  slowers.’  I  can’t  stress  this  strongly  enough  (p.  36). 
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FIGURE  4.  AVERAGE  COST  OVERRUNS  BY  CONTRACT  TYPE 
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In  an  interview  between  the  author  and  Wayne  Abba,  a  respected  analyst 
at  the  Office  of  the  Under  Secretary  of  Defense  (Acquisition),  Abba  com¬ 
mented  that  adverse  trends  can  be  reversed  if  management  pays  attention 
to  them  (W.  Abba,  personal  communication,  1992).  Until  contractors  and 
program  offices  are  willing  to  support  and  advance  realistic  assessments  of 
a  program's  status,  the  attention  and  expertise  of  upper-level  management 
is  postponed,  undoubtedly,  in  the  long  run,  to  the  detriment  of  the  program 
and  nation.  The  famous  economist  Keynes  once  stated  that,  in  the  long  run. 
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we  are  all  dead  (Homgren  &  Foster,  1991).  Postponing  or  hiding  adverse 
information  about  a  program  may  be  an  effective  short-run  strategy;  but,  in 
the  long  run,  it  could  result  in  the  cancellation  of  the  program. 


FIGURE  5.  AVERAGE  COST  OVERRUNS  BY  CONTRACT  PHASE 
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FIGURE  6.  AVERAGE  COST  OVERRUNS  BY  TYPE  OF  WEAPON  SYSTEM 
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FIGURE  7.  AVERAGE  COST  OVERRUNS  BY  MANAGING  SERVICE 
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Endnotes 

1  Responses  from  an  interview  with  J.  J,  Welch,  which  appeared  in  the  Acquisition 
Policy  Letter  91M-005  dated  April  8,  1991. 

Compliance  to  C/SCSC  is  required  on  significant  contracts  and  subcontracts 
within  all  acquisition  programs.  Significant  contracts  are  research,  development, 
test  and  evaluation  contracts  with  an  estimated  cost  of  $60  million  or  more  (in 
fiscal  1990  constant  dollars)  or  procurement  contracts  with  an  estimated  cost  of 
$250  million  or  more  (fiscal  year  1990  constant  dollars)  (Department  of  Defense, 
1991,  p.  ll-B-2). 
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a  Systemic  Fiscal 

in 

Defense  Planning 

(  Leland  G.  Jordan 


Defense  planning  and  budgeting  increase  national  security  costs  by  signifi¬ 
cantly  overestimating  available  future  resources.  An  analysis  of  Department 
of  Defense  out -year  resource  estimates  over  a  period  of  20  years  and  six 
administrations— the  first  econometric  analysis  of  budgeted  and  realized 
resources  in  defense— demonstrates  that  an  optimistic  bias  has  spanned 
administrations  and  appears  to  be  a  systemic  characteristic  rather  than  a 
political  one.  The  result  has  significant  implications  for  reduction  of  defense 
costs  without  loss  of  capability. 


Keywords:  economic  forecasting,  fiscal  policy,  shortfalls  in  resources,  political  and  biased  views 
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Systemic  Fiscal  Optimism 
in  Defense  Planning 

Some  analysts  have  suggested  that  fiscal  optimism  in  defense  plan¬ 
ning  and  budgeting  results  in  less  defense  than  could  have  been  achieved 
given  the  resources  available.  That  is,  fiscal  optimism  results  in  less  bang 
for  the  buck,  rather  than  the  more  bang  for  the  buck  traditionally  sought  by 
the  Department  of  Defense  (DoD).  Programs  established  under  a  projected 
fiscal  regime  with  more  resources  than  later  are  realized  may  become  unaf¬ 
fordable  under  the  tighter  resource  constraint.  Unaffordable  means  that  the 
budget  is  not  sufficient  to  carry  out  the  program  at  the  rate,  at  the  unit  cost, 
and  in  the  quantities  originally  programmed  and  planned. 

The  traditional  issue  of  weapons  systems  cost  growth  and  the  issue  of 
DoD’s  consistent  forecast  that  it  will  receive  significantly  more  budget¬ 
ary  resources  than  it  actually  receives  are  not  separate.  The  gap  between 
planned  and  realized  budgetary  resources  is  the  predominant  cause  of 
weapons  systems  cost  growth. 

Franklin  Spinney  addressed  the  problem  of  cost  growth  and  fiscal  optimism 
in  the  early  1980s.  His  analysis  was  not  well  received  within  DoD,  although 
it  achieved  sufficient  notoriety  outside  DoD:  He  was  pictured  on  the  cover 
of  Time  magazine  (Isaacson,  1983).  He  addressed  the  force  structure  and 
unit  cost  problems  that  result  from  optimistic  assumptions  about  the  cost 
progress  curves1  and  the  reluctance  to  terminate  systems  that,  although 
well  along  in  development  or  production,  appear  unaffordable  given  the 
resources  actually  appropriated  (Spinney,  1980).  Spinney  did  not  address 
how  DoD  consistently  gets  into  the  position  of  not  having  enough  resources 
to  complete  what  it  has  started. 

Gansler  approaches  the  issue  through  the  effects  on  weapons  system  costs 
and  on  strategy  and  the  ability  to  support  strategy  (Gansler,  1989,  chap.  5). 
Focusing  primarily  on  management  within  DoD  and  on  the  interface  with 
industry,  Gansler  addresses  “optimistic  planning,”  but  does  not  directly 
address  the  source  of  fiscal  overoptimism.  The  Packard  Commission 
identified  the  problem  of  optimistic  planning  and  recommended  some 
improvements,  but  did  not  present  an  analysis  demonstrating  the  per¬ 
sistence  of  the  phenomenon  across  time  and  administrations  (Packard, 
1986).  Efforts  to  assess  the  dollar  effect  of  optimistic  planning  have  been 
rare  and  have  not  been  published  in  the  academic  press.  For  example,  Rolf 
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Clark’s  papers,  prepared  under  the  auspices  of  the  DoD’s  Defense  Systems 
Management  College  and  circulated  within  the  DoD,  were  not  published  in 
peer-reviewed  journals  (Clark,  1990a,  1990b). 

This  article  provides  an  assessment  of  the  quality  of  the  defense  out -year 
resource  forecasts  from  a  system  perspective,  identifies  the  source  of  fore¬ 
cast  errors,  and  draws  implications  about  their  costs  and  the  potential  for 
improving  the  forecasts.  Its  broader  purpose  is  to  identify  the  nation’s  out- 
year  budgeting  practices  as  an  important  area  of  research  in  which  analysts 
can  contribute  significantly  to  the  national  welfare.  Budgeting,  whether  for 
next  year  or  longer  periods,  is  an  accountancy  function  directly  affecting 
management;  it  should  be  addressed  with  the  same  rigor  as  is  applied  to  stock 
price  movements,  earnings  forecasts,  and  the  effects  of  revised  standards. 
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This  analysis  is  based  on  the  following  axioms.  If  one  plans  to  have  sig¬ 
nificantly  more  resources  than  become  available  then  it  should  not  be 
surprising  if  the  plans  are  unaffordable.  The  planner  should  learn  from  such 
experience  and  begin  to  estimate  better  the  future  resources.  We  should  not 
expect  a  perfect  forecast,  but  should  expect  the  quality  of  the  forecasts  to 
improve  over  time. 


Forecast  accuracy  is  especially  important  for  national  defense  when 
erroneous  forecasts  contribute  to  a  lesser  capability  than  could 
have  been  obtained  at  the  realized  resource  level. 


The  analysis  presented  here  concludes  that  the  defense 
planning  and  budgeting  system  is  optimistically  biased 


jiA 


and  that  the  bias  has  spanned  several  administrations. 
Nonetheless,  out -year  forecasts  have  been  signifi¬ 
cantly  better  under  some  administrations  than  under 
others.  Those  administrations  having  demonstrated 
the  greatest  bias  in  their  real  growth  projections 
also  have  experienced  the  greatest  shortfalls  in 
resources,  implying  the  greatest  impact  on  man¬ 
agement.  A  proportion  of  the  forecast  error  can  be 
reduced  and  improvements  (discussed  below)  can 
be  instituted. 


Neither  this  analysis  nor  those  cited  suggest  that  opti¬ 
mistic  planning  results  from  malicious  intent.  Rather, 
it  is  the  result  of  a  highly  complex  system  that  does  not 
function  as  intended. 


ip 


*  % 


^  o  ^  £. 

Optimism  is  defined  as  a  form  of  the  bias  discussed  in  the  con¬ 
ceptual  statements  of  the  Financial  Accounting  Standards  Board 
(FASB).  Were  we  able  to  place  a  probability  distribution  on  the  fiscal 
projections  of  the  defense  budget,  we  would  find  that  those  projections  con¬ 
sistently  are  greater  than  the  expected  value.  No  attempt  is  made  to  assign 
the  causes  of  the  bias  to  the  elements  of  measurer  bias  or  measurement 
bias.  Measurer  bias  results  when  the  measurer  misapplies  the  measure¬ 
ment  methodology.  Such  misapplication  may  derive  from  lack  of  skill  or 
lack  of  integrity,  or  both.  Measurement  bias  results  from  inadequacy,  or 
lack  of  validity,  of  the  measurement  instrument  or  method.  However  much 
the  resultant  bias  may  originate  in  each  of  those  two  causes,  it  remains  a 
systemic  characteristic  of  the  national  security  planning  and  budgeting 
system  (FASB,  1985). 
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The  Impact  on  Military  Effectiveness 


Planning  for  more  resources  than  become  available  results  in  program¬ 
ming  a  larger  force  and  more  investments  than  can  be  supported.  The  defense 
literature  has  noted  the  effects  of  that  discrepancy.  Kevin  Lewis,  in  “The 
Discipline  Gap  and  Other  Reasons  for  Humility  and  Realism  in  Defense 
Planning,”  concludes  that  the  likelihood  of  the  DoD’s  planned  program 
achieving  its  planned  effectiveness  is  small  (Lewis,  1994).  It  is  impor¬ 
tant  to  recognize  that  Lewis  has  in  mind  the  military  effectiveness 
of  the  forces  that  result  from  the  plans.  Jacques  S.  Gansler  deals 
with  the  effects  on  weapons  system  costs  and  on  strategy  and 
the  ability  to  support  strategy  (Gansler,  1989).  Spinney  also 
has  addressed  these  effects  (Spinney,  1996). 


In  defense  planning,  the  mix  and  deployment  of 
forces  is  optimized  within  the  expected  resource 
constraints.2  The  mix  of  forces  varies  as  a  function 
of  the  total  financial  resources  available.  For  exam¬ 
ple,  a  specialized  aircraft  or  other  weapons  system 
may  be  effective  and  affordable  only  if  it  exists  in 
the  force  in  some  minimum  quantity.  Fielding  of 
the  system  requires  development  of  doctrine  and 
tactics  and  also  the  training  of  the  forces  and  the 
commanders.  In  the  highly  integrated  modern  battle¬ 
field,  development  and  management  of  compatibility 
with  the  associated  forces  also  is  required.  Clearly,  it 
could  be  ineffective  and  cost-prohibitive  to  do  all  those 
things  for  a  single  aircraft,  especially  if  some  backup  weap¬ 
ons  system  was  required  in  the  event  that  single  aircraft  were 
lost.  At  some  point,  the  cost  effectiveness  of  a  specialized  system, 
available  in  a  minimum  quantity,  is  less  than  the  cost  effectiveness  of 
the  alternative  multipurpose  weapons  system. 


Decisions  to  produce  a  special-purpose  weapons  system  or  the  alternative 
multipurpose  system  are  made  on  the  basis  of  projected  resources.  Even 
once  it  becomes  clear  that  resource  projections  were  optimistic,  revers¬ 
ing  such  decisions  is  difficult.  The  difficulties  arise  from  the  added  costs 
incurred  by  a  termination,  both  economic  and  psychological,  and  from  the 
time-lag  that  would  be  incurred  in  developing  the  multipurpose  system.  In 
fact,  that  time-lag  may  preclude  fielding  of  the  alternative  capability  soon 
enough  to  counter  the  threat.  Thus,  the  ability  to  repair  a  bad  decision  in 
response  to  near-term  information  about  resource  availability  is  limited. 
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Given  the  earlier  decisions,  made  on  the  basis  of  optimistic  resource  projec¬ 
tions,  the  best  possible  defense  program  may  be  significantly  less  effective 
than  would  have  been  possible  had  the  earlier  decisions  been  made  in  the 
context  of  realistic  resource  constraints.  That  situation  is  modeled  below: 

Let  E  (year,  resources,  period)  represent  the  maximum  effectiveness  of  the 
defense  program  resulting  from  decisions  made  in  year  i,  given  multiyear 
projected  resource  constraint),  and  serving  in  the  future  period  k.  The 
period  may  be  a  specified  Future  Years  Defense  Program  (F  YDP)3  period  or 
some  longer  time  span  (such  as  a  decade).  Then,  the  maximum  effectiveness 
of  a  defense  program,  given  resource  constraint )  is 

k)  (1) 

and  the  maximum  effectiveness  of  the  next  year’s  defense  program,  cover¬ 
ing  the  effectiveness  of  the  next  year’s  defense  program,  covering  the  same 
period  k,  but  with  a  revised  resource  constraint^  <  jv  is 

E(i  +  1,  j2,  k)  <  E(i,  jv  k).  (2) 

Some  observers  have  identified  the  revised  planning  that  results  from  cor¬ 
rection  of  E(i,  j ,  k )  to  achieve  E(i  +  1,  j  k)  as  the  source  of  the  acquisition 
turbulence  so  roundly  condemned  by  the  Packard  Commission  (1986). 
Clearly,  if  the  effectiveness  decline  applies  to  the  next  year’s  program,  it  also 
applies  to  the  i  +  n  program  where  n  is  an  integer  greater  than  one  and  less 
than  some  integer  representing  the  time  to  develop  and  field  an  improved 
mix  of  forces. 

Because  the  time  to  develop  and  field  a  weapons  system  is  at  least  10  years, 
the  effectiveness  decline  persists  for  about  that  same  period. 


Other  Analyses  of  Planning  Bias 

The  idea  that  a  bias  in  planning  may  exist  is  not  new.  Henri  Thiel  (1971) 
discusses  the  measurement  of  such  bias  and  offers  several  examples  of  sys¬ 
temic  bias.  His  discussion,  because  it  uses  Dutch  national  forecasts  as  an 
illustrative  case,  establishes  the  relevance  of  that  technique  to  the  analysis 
presented  here.  J.  Chapman  (1981)  applied  Thiel’s  technique  to  assessment 
of  the  accuracy  of  revenue  forecasts  by  California  cities  before  and  after  the 
passage  of  Proposition  13.  He  found  a  tendency  toward  underestimation  of 
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revenues  both  before  and  after  passage  of  Proposition  13.  Chapman’s  find¬ 
ings  are  not  directly  relevant  to  this  analysis,  but  his  application  of  Thiel’s 
technique  is. 


Alan  Auerbach,  an  economist  at  the  University  of 
California,  Berkeley,  commented,  “I’ve  become 
convinced  that  there’s  a  pervasive  tendency  towards 
overoptimism  in  both  agencies” 


Allusions  to  bias  in  national  forecasts  in  the  United  States  are  not  unusual. 
For  example,  D.  Sessel  (1995)  quotes  comments  by  two  well-known  observ¬ 
ers  on  the  White  House  and  Congressional  Budget  Office  (CBO)  forecasts. 
Former  CBO  Director  June  O’Neill  said,  “The  history  over  the  past  20  years 
is  that  both  of  us  are  too  optimistic.”  Alan  Auerback,  an  economist  at  the 
University  of  California,  Berkeley,  commented,  “I’ve  become  convinced 
that  there’s  a  pervasive  tendency  towards  overoptimism  in  both  agencies” 
(Sessel,  1995).  In  Affording  Defense,  Gansler  (1989,  chap.  5)  refers  to  “opti¬ 
mistic  planning.”  One  of  the  threads  of  his  analysis  is  the  effect  of  planning 
for  a  greater  financial  resource  than  becomes  available.  Gansler  is  unusual 
in  his  recognition  of  the  adverse  effects  of  such  optimism. 

The  existence  of  such  a  systemic  bias  is  relevant  to  other  organizations,  both 
public  and  private,  and  knowledge  about  the  detection  and  correction  of  such 
biases  would  be  an  important  contribution  to  the  knowledge  about  manag¬ 
ing  complex  public  and  private  organizations.  The  magnitude  of  the  effect  on 
other  organizations  probably  is  related  positively  to  their  planning  horizon. 


Bias  in  Defense  Planning 

Data 

For  this  study  we  examined  data  for  a  period  of  20  years:  fiscal  year 
1975  through  fiscal  year  1995.4  Planned  resource  levels  were  compared  to 
the  actually  available  resource  levels  for  the  administrations  of  Presidents 
Gerald  Ford,  Jimmy  Carter,  Ronald  Reagan,  George  Bush,  and  for  President 
Bill  Clinton  through  fiscal  year  1995. 
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The  projected  fiscal  resources  against  which  plans  were  constructed  con¬ 
sistently  exceeded  the  fiscal  resources  that  actually  became  available.  The 
situation  is  portrayed  graphically  in  Figure  1.  That  figure  presents  the 
actual  and  planned  data  for  President  Reagan’s  second  administration, 
1985  through  1988. 

The  bars  in  the  chart  show  the  resource  levels  for  each  year  of  the  DoD’s 
planning  period.  Because  a  new  planning  period  begins  yearly,  the  bars  for 
each  year  represent  plans  from  several  prior  years.  The  line  represents  the 
funding  appropriated  by  the  Congress,  the  fiscal  resources  that  was  realized 
resource  that  was  realized.  Clearly,  the  plans  of  each  administration  extend 
into  the  subsequent  administrations  and,  just  as  clearly,  are  revised  by  those 
subsequent  administrations. 


FIGURE  1.  PLANNED  VERSUS  REALIZED  REAL  GROWTH, 
FISCAL  YEARS  1985-88 


300 


1984  1985  1986  1987  1988  1989  1990  1991  1992 


FISCAL  YEARS 


FY85  Plan  MFY86Plan  MFY87Plan  HFY88Plan  Realizations 


In  order  to  remove  the  effects  of  inflation,  real  dollar  levels  indexed  on  fiscal 
year  1974  are  plotted.  The  DoD  deflators  were  applied  to  the  actual  appro¬ 
priations.  Those  projected  at  the  time  of  each  plan  were  used  to  deflate  the 
resource  projections,  and  then  were  linked  to  the  same  deflators  that  were 
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applied  to  the  series  of  actual  appropriations.  Thus,  each  year’s  resource 
levels,  projected  and  actual,  were  restated  in  the  same  dollars  and  then 
indexed  on  fiscal  year  1974. 

Methodology 

Spinney  (1992)  used  a  primarily  graphical  analysis  in  his  presentations, 
accompanied  by  discussion.  Figure  1  similarly  portrays  the  data.  Graphical 
portrayals  provide  an  intuitive  feel  for  the  situation,  but  they  do  not  support 
conclusions  about  the  underlying  causes  of  the  forecasting  errors. 

In  Applied  Economic  Forecasting,  Thiel  (1971,  p.  32)  develops  a  method 
for  analyzing  the  adequacy  of  economic  forecasts.  Thiel  decomposes  the 
squared  error  of  the  forecast  into  coefficients  related  to  the  sources  of  the 
forecasting  error. 

Our  analysis  is  based  on  real  growth  rates,  projected  and  actual,  to  remove 
the  effects  of  inflation  and  also  because  the  proj  ection  methodology  used  in 
the  DoD  is  based  largely  on  assumptions  of  future  real  growth.  The  analysis 
uses  the  natural  logarithms  of  the  real  growth  rates.  Their  use  ensures  that 
the  levels  in  years  t  and  tz  are  the  same  if  the  log  changes  in  those  years  are 
equal  but  of  opposite  sign  (Thiel,  pp.  47-50). 

Sources  of  the  Projection  Errors 

Thiel’s  coefficients  are  derived  from  the  sum  of  the  squared  errors  as 
shown  below  in  Figure  2.  The  coefficients  represent  bias,  variance,  and 
covariance,  respectively. 


FIGURE  2.  THE  SUM  OF  THE  SQUARED  ERRORS 


—I  (P.-A  )2  =  (P-A)2  +  (S  -S  ) 2  +2  (  1  -r)S  S 

n  v  I  i  7  v  7  v  p  a  /  v  'pc 
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Analysis  of  the  Data 

Table  1  presents  data  about  the  frequency  of  the  forecasting  errors.  Table 
2  presents  the  coefficients  and  is  followed  by  a  discussion  of  their  meaning. 

As  Table  1  shows,  the  real  growth  rate  used  in  DoD’s  resource  projections 
exceeded  the  real  growth  rate  realized  in  the  amounts  appropriated  in  66  of 
94  fiscal  years  (70  percent  of  the  projections).  The  effects  of  inflation  have 
been  removed  from  both  the  resource  projections  and  the  appropriated 
amounts.  The  optimistic  tendency  (70  percent  of  the  projections  exceeded 
the  actual  appropriations,  in  real  dollars,  vice  the  approximately  50  percent 
in  an  unbiased  system),  therefore,  is  not  a  result  of  the  difficulty  of  forecast¬ 
ing  inflation  rates. 

Bias 

The  bias  proportion  represents  deviations  in  central  tendency.  It  shows 
the  proportion  of  the  root  mean  square  error  that  results  from  the  differ¬ 
ence  between  the  mean  of  the  predictions  and  the  mean  of  the  realizations. 
Positive  values  for  the  difference  in  the  means  of  the  predicted  and  realized 
values  indicate  that,  on  the  average,  higher  real  growth  rates  are  projected 
than  are  realized. 

In  five  of  the  six  administrations  the  mean  prediction  exceeded  the  mean 
realization.  The  importance  of  that  bias  is  indicated  by  the  bias  propor¬ 
tions  in  Table  2.  In  each  of  President  Reagan’s  administrations,  about  75 
percent  of  the  error  in  projections  derived  from  optimism  about  how  much 
Congress  would  appropriate.  In  President  Bush’s  administration,  about 
49  percent  of  the  projection  error  resulted  from  an  upward  bias.  About 
27  percent  of  the  projection  error  in  President  Clinton’s  first  two  years 
resulted  from  overly  optimistic  projections.  President  Ford’s  administra¬ 
tion  exhibited  very  little  bias.  About  0.3  percent  of  his  projection  error 
resulted  from  general  overoptimism. 


TABLE  1.  FREQUENCY  OF  FORECASTING  ERRORS 


Number  of  periods  forecast 

94 

Forecast  real  growth  rate  exceeded  actual  rate 

66 

Actual  rate  exceeded  forecast 

28 
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TABLE  2.  INEQUALITY  PROPORTIONS  jj 

Administration 

Bias 

Variance 

Covariance 

Ford 

.00322 

.23899 

.75779 

Carter 

.33380 

.32189 

.34431 

Reagan  1 

.75237 

.03787 

.20976 

Reagan  II 

.75249 

.00044 

.24707 

Bush 

.48722 

.05170 

.46109 

Clinton 

.26872 

43326 

.29802 

In  contrast,  President  Carter’s  administration  exhibited  a  bias  below  what 
the  Congress  appropriated,  accounting  for  about  one-third  of  the  projec¬ 
tion  error. 


Variance 

The  variance  proportion  is  zero  only  if  the  standard  deviations  of  the 
projected  and  realized  real  growth  rates  are  the  same.  As  Table  3  indi¬ 
cates,  for  the  administrations  of  Presidents  Ford,  Carter,  and  Bush  and  for 
President  Reagan’s  first  administration,  the  variance  of  the  realizations 
exceeded  the  variance  of  the  projections.  For  each  of  those  administra¬ 
tions,  the  projected  real  growth  rate  fluctuated  less  from  year  to  year  than 
did  the  achieved  real  growth  resulting  from  congressional  appropriations. 
For  Presidents  Reagan  and  Bush,  this  difference  in  consistency  contributed 
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only  about  3.8  percent  and  5.2  percent,  respectively,  of  their  projection 
error,  making  that  source  relatively  unimportant  compared  to  the  effect  of 
the  upward  bias  in  central  tendency.  During  the  Carter  and  Ford  admin¬ 
istrations,  the  difference  in  consistency  was  relatively  more  important, 
contributing  32  percent  and  24  percent,  respectively,  of  the  projection  error. 

Conversely,  in  President  Reagan’s  second  administration  and  in  the  first  two 
years  of  President  Clinton’s  administration  the  projections  have  been  less 
tightly  distributed  than  have  the  congressional  appropriations. 


TABLE  3.  MEANS, 

STANDARD  DEVIATIONS,  AND  CORRELATIONS 

Administration 

P-A 

S  “  S 

P  3 

r 

Ford 

.0028 

-.0238 

-.0736 

Carter 

-.0294 

-.0289 

.0820 

Reagan  1 

.0522 

-.0117 

.7325 

Reagan  II 

.4934 

.0019 

.0937 

Bush 

.0436 

-.0141 

.1458 

Clinton 

.0123 

.0156 

.8524 

One  might  hypothesize  that  the  pattern  of  the  variance  relationship  indi¬ 
cates  that  Presidents  Ford,  Carter,  Bush,  and  Reagan  (in  his  first  term)  had 
a  better-defined  vision,  or  at  least  a  firmer  vision,  for  the  national  security 
than  did  the  Congress.  Such  a  hypothesis  would  accept  year-to-  year  consis¬ 
tency  in  appropriations  as  a  proxy  for  a  consistent  vision.  A  full  examination 
of  that  hypothesis,  however,  is  beyond  the  scope  of  this  paper;  for  the  present, 
we  leave  it  for  others  to  address.  As  one  reviewer  noted,  however,  it  might  be 
addressed  through  an  analysis  of  the  concurrent  resolutions  on  the  budget.5 

Covariance 

The  covariance  proportion  is  zero  only  if  the  coefficient  of  correlation 
is  1.  As  indicated  in  Table  3,  the  directional  agreement,  the  correlation, 
between  the  administration’s  real  growth  projections  and  the  congres¬ 
sional  appropriations,  has  been  highest  in  President  Reagan’s  first  term 
and  in  President  Clinton’s  first  two  years.  President  Ford  and  the  Congress 
moved  in  opposite  directions.  The  correlations  in  President  Carter’s  admin¬ 
istration,  President  Reagan’s  second  administration,  and  President  Bush’s 
administration  are  all  positive,  but  quite  low.  The  difference  in  correla¬ 
tion  contributed  relatively  significantly  to  the  projection  error  in  the  Ford 
administration,  less  so  in  the  Bush  administration,  and  progressively  less 
so  in  the  Carter,  Clinton,  and  Reagan  administrations. 
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The  preceding  analysis  addressed  the  sources  of  the  projection  error  on  a 
relative  basis.  If  the  projection  error  is  small,  then  the  importance  of  a  rela¬ 
tively  large  proportional  contribution  also  is  small.  Thus,  it  is  important  to 
address  the  size  of  the  projection  errors.  Did  the  administrations  have  simi¬ 
lar  projection  errors,  or  did  some  administrations  experience  notably  large 
projection  errors?  What  was  the  source  of  any  larger-than-typical  errors? 

Sizes  of  the  Projection  Errors 

Table  4  presents  the  average  sizes  of  the  projection  errors  as  a  percent¬ 
age  of  the  planned  resource  level;  that  is,  as  a  percentage  of  the  projection. 
Importantly,  for  Table  4,  the  calculation  is  based  on  the  planned  resource 
level  (in  constant  dollars),  not  on  the  year-to-year  real  growth  rates,  and 
is  not  represented  logarithmically.  The  resource-level  base  portrays  the 
effect  on  program  management  better  than  do  the  calculations  based  on 
year-to-year  rates. 


TABLE  4.  SIZE  OF  RESOURCE  SHORTFALLS  ji 

Administration 

Mean  Shortfall  as  Percent  of 
Planned  Resource  Level 

Ford 

-5.3 

Carter 

+10.9 

Reagan  1 

-13.4 

Reagan  II 

-19.5 

Bush 

-11.1 

Clinton 

-2.5 

Note.  A  minus  sign  indicates  available  resources  were  less  than  planned.  The 
comparison  is  across  the  periods  projected  during  each  Presidential  term. 


It  is  the  error  in  projecting  year-to-year  real  growth  rates  that  causes  the 
resource  shortfalls  and  that  error  is  an  accurate  portrayal  of  the  overopti¬ 
mism.  The  overly  optimistic  projection  of  future  resources  derives  from  the 
overly  optimistic  projections  of  real  growth.  Nonetheless,  once  resources  are 
realized,  it  is  the  resource  quantity  that  constrains  management  of  opera¬ 
tions  and  investment.  Hence,  the  importance  of  those  shortfalls  is  better 
measured  as  a  function  of  the  resource  levels.  Note  that  measurement  using 
the  resource  levels  makes  each  year’s  error  dependent  on  the  cumulative 
effect  of  the  prior  years’  errors,  as  it  in  fact  is. 
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Of  the  six  administrations,  only  President  Carter’s  projected  less  in 
resources  than  were  realized.  It  is  enlightening,  however,  to  look  at  the  tim¬ 
ing  and  circumstance  of  those  in-excess-of-projected  realizations.  During 
President  Carter’s  tenure,  the  Congress  appropriated  an  average  of  4.9 
percent  more  than  President  Carter  requested.  The  Carter  administration 
projections  for  his  post -tenure  years,  fiscal  years  1982  through  1986,  were 
significantly  less  than  Congress  appropriated  for  those  years.  President 
Carter’s  plans  for  those  post-tenure  years  were  overfunded  by  an  average 
of  13.8  percent  of  those  plans. 

Thus,  President  Carter’s  average  resource  overrun  of  10.9  percent  com¬ 
pared  to  his  out -year  projections  can  be  attributed  largely  to  President 
Reagan’s  military  buildup.  The  Carter  administration’s  bias  to  the  low  side 
of  those  realizations  appears  to  be  a  result  of  a  changed  national  security 
policy  and  perception. 

Conversely,  the  existence  of  overfunded  plans  during  President  Carter’s 
tenure  confirms  that  it  is  possible  for  a  President  to  overcome  the  systemic 
fiscal  overoptimism  of  the  defense  establishment.  Of  the  six  administra¬ 
tions  analyzed,  only  the  Carter  administration  presented  requests  to  the 
Congress  that  were  less  than  the  amount  ultimately  appropriated  by  the 
Congress.  If  we  conclude  that  the  defense  budget  process  includes  a  systemic 
overoptimism,  then  we  are  led  to  conclude  that  President  Carter  managed 
to  overcome  that  systemic  bias. 

Spinney  (1996)  offers  a  description  of  the  pressures  to  increase  budget 
allocations  that  the  defense  establishment  can  place  on  a  President.  His 
recounting  of  the  pressures  and  maneuvering  leading  to  the  1996  increases 
in  the  future-years  program  provides  considerable  insight  into  the  dif¬ 
ficulties  a  President  faces  in  overcoming  defense’s  tendency  to  optimistic 
out-year  fiscal  projections. 

Correlation  of  Projection-Error  Size  and  the  Bias  Coefficient 

Those  administrations  having  the  largest  projection  errors,  as  measured 
by  Table  4,  also  exhibit  the  largest  bias  coefficients.  Consider  Table  5.  The 
apparently  high  correlation  is  confirmed  by  a  Spearman  Rank  correlation 
test.  That  test,  yielding  a  rank  correlation  coefficient  of  0.94,  is  significant 
on  a  one-sided  test  with  a  type  I  error  of  0.02. 
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TABLE  5. 

CORRELATION  OF  ERROR  SIZE  AND  BIAS 

Administration 

Absolute  Value  of  Mean 
Shortfall  as  Percent  of 
Planned  Resource  Level 
<%> 

Bias  Coefficient 
(%) 

Reagan  II 

19.5 

75.25 

Reagan  1 

13.4 

75.24 

Bush 

11.1 

48.72 

Carter 

10.9 

33.38 

Ford 

5.3 

3.22 

Clinton 

2.5 

26.87 

Thus,  over  the  past  20  years,  those  administrations  that  exhibited  signifi¬ 
cant  bias  (optimistic  or  pessimistic)  in  their  resource  projections,  tended 
also  to  have  relatively  large  errors  in  their  projections  of  resources. 

Consider  Table  6,  which  is  Table  2  reordered  from  the  largest  to  the  small¬ 
est  projection  error,  except  for  the  Ford  administration.  If  we  accept  that 
President  Ford’s  projection  error  derived  primarily  from  his  directional 
differences  with  the  Congress,  then  the  evidence  becomes  more  persuasive. 


TABLE  6.  INEQUALITY  PROPORTIONS  ORDERED  BY  SIZE  OF 

PROJECTION  ERROR  (EXCEPT  FOR  FORD  ADMINISTRATION) 

Administration 

Bias 

Variance 

Covariance 

Reagan  II 

.75249 

.00044 

.24707 

Reagan  1 

.75237 

.03787 

.20976 

Bush 

.48722 

.05170 

.46109 

Carter 

.33380 

.32189 

.34431 

Clinton 

.26872 

. 43326 

.29802 

Ford 

.00322 

.23899 

.75779 

If  the  bias  coefficient  is  large,  then  the  average  predicted  change  is  substan¬ 
tially  different  from  the  average  realized  change.  If  bias  remains  a  major 
source  of  error  over  time,  then  the  forecasting  system  is  not  improving. 
That  is  a  serious  error.  The  covariance  error  source  should  not  be  expected 
to  approach  zero.  Were  that  true,  the  line  of  predictions  and  realizations 
would  be  straight.  Such  an  exact  alignment  is  too  much  to  expect  (Thiel, 
1971,  p.  32). 
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Efforts  to  Correct  the  Bias  Error 

Bias  has  been  recognized  as  a  serious  source  of  error  by  an  independent 
commission  and  within  the  Department  of  Defense  (Gansler,  1989;  Packard, 
1986;  Spinney,  1996).  Recognition  within  the  Department  is  difficult  to 
document  because  internal  DoD  management  and  financial  management 
policy  analyses  are  not  publicly  available.  Nonetheless,  there  has  been 
sufficient  occasional  recognition  of  fiscal  overoptimism  as  a  management 
problem  to  support  the  conclusion  that  the  professional  career  staff  was 
aware  of  it  and  of  its  deleterious  effects  (Clark,  1990a;  Clark,  1990b;  Jordan, 
1990;  Lewis,  1994). 

The  Packard  Commission  (1986)  focused  intensively  on  the  tendency  to 
overestimate  the  future  resources  as  a  serious  management  problem.  That 
Commission’s  report,  together  with  pressure  from  career  executives,  fostered 
a  limited  recognition  of  the  need  to  improve  the  forecasting  of  resources. 


Conclusions  and  Recommendations 

A  tendency  exists  for  the  Defense  Department  to  project  the  availability 
of  significantly  more  resources  than  become  available.  Historically,  those 
administrations  having  demonstrated  the  greatest  bias  in  their  real  growth 
projections  also  have  experienced  the  greatest  shortfalls  in  resources. 
Hence,  those  administrations  having  demonstrated  the  greatest  bias  in  their 
real  growth  projections  also  most  seriously  handicapped  program  manag¬ 
ers.  Projecting  significantly  more  resources  than  become  available  directly 
affects  force  mix  and  capability.  The  force-mix  optimization  studies  used  in 
programming  decisions  incorporate  a  resource  constraint. 

The  existence  of  optimistic  bias  has  spanned  administrations.  It  contin¬ 
ues  despite  changes  of  administrations— whether  the  political  party  of 
the  incoming  administration  is  the  same  or  changes.  It  appears,  therefore, 
that  the  bias  results  from  some  characteristic  of  the  defense  management 
system;  it  is  a  systemic  phenomenon.  So  it  appears  reasonable  to  conclude 
that  reducing  the  optimistic  bias  will  require  changes  to  the  planning  and 
budgeting  system.  In  undertaking  such  changes,  it  is  important  to  recognize 
that  bias  reduction  is  the  goal,  not  elimination  of  the  projection  error. 

There  clearly  is  room  for  improvement  in  the  Defense  planning  and  budget¬ 
ing  system.  The  analysis  in  this  article  is  empirical.  It  establishes  existence 
of  a  systemic  bias  in  one  the  nation’s  major  accounting  and  budgeting 
systems.  Gansler  (1989)  and  Clark  (1990a,  1990b)  each  have  identified 
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significant  costs  arising  from  budget  turbulence  in  DoD.  The  systemic  bias 
identified  here  is  a  source  of  that  turbulence.  It  seems  reasonable  to  suggest 
that  other  analysts  could  contribute  significantly  to  the  national  welfare 
via  rigorous  development  of  improved  forecasting  methods  that  would  be 
unbiased.  A  broad  proposal  for  such  research  is  outlined  below. 

Changes  in  the  planning  and  budgeting  system  to  reduce  opti¬ 
mistic  bias  should  be  based  on  a  review  that  identifies  the 
decision  points  and  techniques  of  the  system.  Techniques 
include  the  modeling  and  projection  methodology;  for  example, 
regression  analysis,  auto  regressive  integrated  moving  aver¬ 
age  (ARIMA),  or  dynamic  economic  models.  Decision  points 
are  those  places  in  the  process  where  out -year  assumptions  are 
made.  Examples  of  these  are  whether  DoD  will  receive  a  greater 
or  smaller  share  of  the  U.S.  budget,  whether  the  U.S.  budget  will 
increase  or  decrease,  and  the  size  of  the  applicable  growth  rates. 

Because  the  analysis  identifies  a  period  in  which  the  systemic 
bias  was  corrected,  a  comparison  of  that  period  to  other  periods 
appears  potentially  fruitful.  The  first  step  in  such  research  might 
be  structured  interviews  with  senior  officials  and  analysts  who 
played  key  roles  in  the  planning  and  budgeting  process  under  the 
Carter  administration  and  other  administrations. 

Three  sources  of  projection  error  were  identified:  bias,  variance, 
and  covariance.  It  is  reasonable  to  expect  that  forecasting  systems 
should  exhibit  the  ability  over  time  to  diminish  the  bias  source. 

Not  to  do  so  indicates  lack  of  continuing  improvement  in  the  fore¬ 
casting  system.  The  time  trend  of  bias  errors  does  not  indicate  any 
systemic  improvement.  From  a  system  perspective,  the  national 
defense  planning  system  is  not  functioning  as  it  should.  The  vari¬ 
ance  error  source  appears  to  result  from  the  relative  consistency 
of  the  administration’s  vision  of  the  national  defense  versus 
the  consistency  of  the  Congress’s  vision.  Testing  and  analysis 
of  that  hypothesis  is  deferred,  but  changes  to  the  forecasting 
system  appear  an  unlikely  way  to  improve  the  correlation  of  the 
Administration’s  and  the  Congress’s  vision  for  national  defense. 

The  covariance  error  source  should  be  expected  to  continue; 
further,  improvements  in  the  forecasting  system  that  reduce  the 
bias  source  almost  surely  will  increase  the  relative  size  of  the 
covariance  error  source. 
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Endnotes 

1  The  optimistic  assumption  is  production  quantity,  not  slope. 

2  Optimization  within  resource  constraints  is  well  established  in  national  security 
planning.  The  techniques  and  theory  were  set  out  30  years  ago  by  Quade  and 
Boucher  (1968).  The  Packard  Commission’s  report  (Packard,  1986)  clearly  reflects 
the  continuation  of  that  practice. 

3  A  Future  Years  Defense  Program  covers  a  specified  6-year  period  for  which 
DoD  plans.  A  new  FYDP  period  starts  each  biennium,  thus  constituting  a  rolling 
coverage  of  the  future. 

4  Data  are  from  DoD  press  releases,  Secretary  of  Defense  Annual  reports  to  the 
Congress,  and  the  National  Defense  Budget  Estimates  series  published  by  the  DoD 
Comptroller.  Data  for  earlier  years  were  not  available.  Although  the  “Historical 
FYDP”  reaches  back  to  fiscal  year  1962,  it  does  not  present  the  original  estimates. 
FYDP  data  are  revised  if  appropriations  change  during  the  year  and  also  to  reflect 
actual  obligations  through  time.  In  addition,  documents  presenting  the  original 
inflation  forecasts  are  not  available  and  such  original  projections  are  necessary  to 
restate  the  out-year  data  in  constant  dollars. 

5  Analyses  of  the  congressional  budget  process  are  in  Joyce  (1996)  and  Shick 
(1996). 
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Resilience— A 

Concept 

Co/  Dennis  J.  Rensel,  USAF  (Ret.) 

Resilience  takes  on  many  definitions  and  ideas  depending 
upon  who  is  speaking.  Taking  this  one  step  further, 
consider  resiliency  as  a  concept  that  provides  a  holistic 
view  of  a  system  or  capability,  just  as  biomedical  indices 
provide  an  indication,  a  concept  of  a  person’s  health. 
This  process  or  concept  of  assessing  one’s  health  can  be 
equated  to  the  assessment  of  the  health  of  a  network  or 
system.  The  hypothesis  is:  resiliency  is  meaningful  in 
the  context  of  holistic  assessments  of  capabilities.  At 
this  level,  comparisons  of  capabilities  or  systems  can 
lead  to  informed  decisions  about  resources,  funding,  and 
tradespaces.  This  article  develops  a  Resiliency  Tier  Matrix 
and  illustrates  how  to  obtain  a  holistic  view  of  resilience 
for  a  capability  or  system. 

Keywords:  resilience,  health,  holistic,  Resiliency  Tier,  Resiliency  Tier 
Matrix,  State  of  Resiliency 
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Resilience  as  aterm  has  as  many  definitions  as  people  who  talk  about  it. 
What  if  resiliency  is  treated  as  a  concept?  How  do  you  measure  a  concept?  In 
reviewing  many  definitions,  “each  [definition] ...  rests  on  one  or  two  essential 
aspects  of  resilience:  continuity  and  recovery  in  the  face  of  change”  (Zolli  & 
Healy,  2012,  p.  7).  A  key  to  the  success  of  any  resiliency  analysis  is  to  fully 
understand  the  level  of  protection  and  tolerance  that  is  acceptable  to  meet 
mission  needs  and  then  to  create  a  strategic  plan  accordingly.  A  true  resil¬ 
ience  measure  is  holistic,  viewing  the  whole  of  a  robust  mission  capability 
and  not  a  sum  of  each  component’s  capability. 


Capitalizing  on  this  holistic  view,  the  resulting  analysis  com¬ 
pares  and  contrasts  various  capabilities  with  different 
conditions,  requirements,  and  operations.  Working  within 
this  tradespace,  analysis  may  lead  to  critical  junctures: 
Capability  vs.  Cost,  Improvements  vs.  New  Development,  or 
Research  and  Development  Investments  vs.  Sustainment. 
Knowing  the  State  of  Resiliency  of  a  system  will  lead  to 
answers  to:  How  can  resiliency  be  improved?  Where  should 
the  next  dollar  go?  And  when  has  a  system  reached  its 
end  of  life?  This  information  can  lead  to  informed 
decisions  and  better  capabilities. 

Effective  resiliency  planning  comes  from  under¬ 
standing  situational  and  mission  needs  before  a 
disastrous  event  occurs.  Developing  a  Resiliency  Tier 
Matrix  would  capture  this  situational  and  mission  aware¬ 
ness.  Resiliency  Tiers  demonstrate  acceptable  tolerance  for  the 
system/capability  to  meet  mission  needs.  A  goal  in  this  entire  process 
is  to  create  a  true  holistic  Resiliency  Index  that  reflects  more  than  each 
functional  component’s  contribution. 


Hypothesis 


The  holistic  analysis  of  resiliency  provides  insight  into  a  capa¬ 
bility  or  system’s  resilient  characteristics  and  provides  a  means 
for  creating  informed  decisions  regarding  funding,  devel¬ 
opment,  deployment,  and  mission  accomplishment. 
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Purpose 

This  article  presents  resiliency  as  a  concept  that  incorporates  many 
other  factors  and  elements  and  develops  a  Resiliency  Tier  Matrix  for 
analysis  purposes. 


Scope 

This  article  portrays  resiliency  as  an  overarching  concept  that  affects 
capabilities  and  systems  differently  depending  upon  the  situation.  It  devel¬ 
ops  a  Resiliency  Tier  Matrix  to  provide  a  holistic  view  of  what  resilience 
means  to  that  capability  or  system.  The  research  was  limited  to  recent  arti¬ 
cles  on  resiliency  and  various  interpretations  of  resilience  and  its  effects. 
The  development  of  the  Resiliency  Tier  Matrix  involves  the  relationships 
between  existing  conditions  and  possible  impacts  to  capabilities  and  sys¬ 
tems.  Use  of  the  matrix  provides  decision  makers  with  knowledge  to  make 
informed  decisions.  This  article  does  not  delve  into  resiliency  associated 
with  people  or  organizations  because  an  abundance  of  literature  already 
covers  the  many  aspects  of  these  two  constructs. 


Discussion 

The  word  resiliency  has  no  universally  accepted  definition.  Many  orga¬ 
nizations  have  coined  more  than  one  definition.  One  of  the  more  accepted 
definitions  is  from  the  Office  of  the  Secretary  of  Defense  (Policy)  (Department 
of  Defense,  2012): 

The  ability  of  an  architecture  to  support  the  functions 
necessary  for  mission  success  with  higher  probability, 
shorter  periods  of  reduced  capability,  and  across  a  wider 
range  of  scenarios,  conditions,  and  threats,  in  spite  of  hos¬ 
tile  action  or  adverse  conditions.  Resilience  may  leverage 
cross-domain  or  alternative  government,  commercial,  or 
international  capabilities,  (p.  12) 

Resilience  is  an  overarching  concept  or  an  umbrella,  which  encompasses 
many  other  concepts,  characteristics,  or  parameters.  Each  may  have  a  major 
impact  at  any  one  time.  This  leads  to  the  basic  question  of  howthe  resiliency 
of  a  capability  can  be  improved.  Many  synergies  and  forces  play  important 
roles.  Turning  to  systems,  resiliency  incorporates  many  other  metrics  and 
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variables.  Figure  1  shows  the  various  parameters  and  techniques  associated 
with  resiliency.  As  a  concept,  no  single  metric  does  resiliency  sufficient  jus¬ 
tice.  When  defining  a  specific  metric,  another  aspect  of  resiliency  surfaces. 
The  first  metric  no  longer  fits  because  the  emphasis  shifted  to  the  next 
aspect  or  dimension. 


FIGURE  1.  RESILIENCY  UMBRELLA 
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Resiliency  as  a  term  applies  to  people,  organizations,  and  items/systems. 
Information  technology  networks,  ecological  systems,  social  environments, 
and  health  conditions  use  the  term.  For  each  of  these  constructs,  risks  come 
from  all  directions:  events,  data  operations,  or  even  missions.  Risks  are  gen¬ 
erally  more  prevalent  during  events  such  as  an  adversarial  attack  or  natural 
disaster  or  even  from  a  series  of  minor  incidents  that  add  up.  Preparation  to 
meet  these  challenges  would  minimize  exposure  and  provide  faster  reac¬ 
tion  times.  One  means  of  minimizing  effects  would  be  to  understand  system 
vulnerabilities.  Many  of  the  ideas  and  concepts  are  taken  from  an  IBM 
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paper  on  Business  Resilience  (IBM,  2009,  p.  5).  Even  though  the  IBM  article 
focuses  on  business  and  business  management,  a  variation  or  derivation  of 
its  resiliency  framework  can  be  extended  to  systems  and  their  environment. 

The  success  of  any  assessment/estimation  is  situational  awareness  of  all 
aspects  of  resiliency.  It  helps  define  the  level  of  protection  and  tolerance  that 
is  acceptable.  Appendix  A  describes  a  Resiliency  Black  Box  and  the  interac¬ 
tions  of  the  various  parameters  in  Figure  1  under  the  Resiliency  Umbrella. 
A  strategic  plan  is  needed  to  meet  mission  resiliency  requirements.  The 
implementation  of  such  a  plan  comes  with  challenges:  (a)  assessing  risk 
vs.  cost  -  what  level  of  vulnerability  is  tolerable?  (b)  viewing  resilience  as 
a  strategic  enabler,  (c)  developing  a  resilience  culture,  (d)  assessing  return 
on  investment  for  resilience  strategies  (IBM,  2009,  p.  7),  and  (e)  linking 
capabilities  to  mission  requirements.  However,  done  correctly,  the  imple¬ 
mentation  could  lead  to  informed  decisions  about  tradespace  and  alternative 
actions  beyond  the  technical  solution. 

Open  literature  discusses  resiliency  techniques.  These  seem  to  fall  into 
three  categories.  The  first  category  is  human  behavioral  practices,  social 
and  societal  impacts  (The  State  of  New  York,  2013,  p.  3),  and  application  to 
systems-of-systems  (Bodeau,  Brtis,  Graubart,  &  Salwen,  2013,  p.  1).  This 
category  is  outside  the  scope  of  this  article.  The  second  category  illustrates 
approaches  through  case  studies  on  how  some  communities  increased  their 
resilience  within  their  environment.  The  third  category  provides  an  engineer¬ 
ing  framework  for  mapping  goals  to  objectives  to  techniques.  Figure  1  depicts 
many  of  these  techniques,  which  lead  into  this  Resiliency  Tier  development. 
The  desired  outcome  is  then  to  develop  innovative  measures  to  enhance  resil¬ 
iency  similar  to  what  the  communities  did  in  the  second  category. 

In  treating  resiliency  as  a  multidimensional  concept,  there  needs  to  be 
a  way  to  characterize  it  and  still  have  some  quantitative  assessment.  An 
analogy  would  be  the  status  of  a  person’s  health,  which  is  multidimensional. 
Numerous  medical  indices  cover  all  aspects  of  health:  temperature,  weight, 
disease  conditions,  muscle  tone,  aging,  etc.  But  when  asked  how  healthy  a 
person  is,  a  general  concept  of  what  all  the  indices  or  parameters  indicate  is 
the  appropriate  answer.  Resiliency  can  adopt  the  same  construct.  If  resil¬ 
iency  of  a  system  equates  to  the  health  of  a  person,  then  maybe  there  should 
be  resiliency  indices  similar  to  health  indices.  Just  like  the  health  hazards 
that  people  experience,  systems  experience  multiple  attacks  on  their  con¬ 
figurations.  A  specific  health  index  addresses  a  specific  health  condition  or 
set  of  related  conditions.  Depending  upon  the  value  and  importance  of  the 
index,  patients  will  spend  their  last  dollar  on  a  remedy.  To  obtain  a  cure. 
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patients  need  to  learn  the  overall  concept  of  their  health.  This  is  where 
assessment  of  the  myriad  of  available  health  indices  is  invaluable  in  deter¬ 
mining  their  state  of  health.  Indeed,  the  decision  may  impact  where  patients 
choose  to  spend  their  health  dollars.  A  similar  analytical  process  can  apply 
to  systems  or  capabilities  and  their  resiliency.  The  assessment  of  these 
various  parameters  or  dimensions  can  determine  a  State  of  Resiliency  and 
would  lead  to  a  holistic  view  of  the  system.  This  type  of  assessment  informs 
budget,  development,  and/or  deployment  decisions. 

There  can  be  many  indices  describing  resiliency,  each  emphasizing  a  dif¬ 
ferent  aspect.  However,  when  asked  how  resilient  a  system  or  capability  is, 
the  answer  should  encompass  the  varied  indications  from  the  set  of  resil¬ 
iency  indices.  If  done  correctly,  this  Resiliency  Index  would  allow  for 
comparisons  of  capabilities  or  systems  within  a  tradespace.  For  purposes 
of  this  discussion,  since  the  relationship  between  systems  and  capabilities 
is  close,  the  rest  of  the  article  will  concentrate  on  systems. 


As  a  management  tool,  the  Resiliency  Tier  Framework 
offers  a  way  to  compare  various  programs,  systems, 
and  capabilities  in  terms  of  potential  tradespace,  cost 
savings,  or  capability  optimization. 

In  reviewing  literature,  we  found  many  articles  that  discussed  metrics 
for  resiliency.  The  Defense  Science  Board  Task  Force  built  a  notional 
dashboard-metric  collection  system  (DoD,  2013,  p.  13).  This  system,  hav¬ 
ing  maturity  levels  and  designed  metrics,  supported  cyber  systems  at  a  very 
detailed  level.  In  contrast,  IBM  developed  a  Resilience  Tier  Framework 
(IBM,  2009,  p.  14).  This  framework  defines  levels  of  resilience  to  match 
business-driven  requirements.  It  spans  all  business  units,  services,  and 
technologies.  It  provides  the  client  a  streamlined  direction  for  building 
a  resilient  architecture.  Ultimately,  a  true  resilience  measure  is  holistic, 
encompassing  the  operations,  technology,  and  culture  of  an  organization.  In 
a  variation  of  the  IBM  model,  the  Resiliency  Tier  Matrix  in  this  article  has 
five  Resiliency  Tiers  ranging  from  Tier  I,  which  is  a  total  disaster,  to  Tier 
V,  which  is  the  gold  standard.  In  this  case,  12  different  indices  are  spread 
across  the  five  tiers  to  assess  the  overall  resiliency  of  a  system. 

Any  military  capability  encounters  numerous  hazards  or  risks  from  all 
directions.  Examples  of  sources  for  these  risks  are  events,  system  fail¬ 
ures,  or  human  error.  These  risks  can  be  minor  or  major  depending  upon 
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the  conditions.  To  minimize  the  effects,  system  users  need  to  be  aware  of 
vulnerabilities  and  have  mitigating  actions  in  place.  Effective  prepara¬ 
tions  and  actions  involve  a  holistic  approach  with  proactive  processes  and 
vigilant  situational  awareness  for  the  unknown  (IBM,  2009,  p.  5).  When 
system  users  develop  this  holistic  view,  an  extensive  analysis  compares  and 
contrasts  various  capabilities,  different  conditions,  environments,  mission 
requirements,  and  operations.  Armed  with  this  view,  decision  makers  can 
make  informed  decisions  regarding  better  capabilities  and  their  use. 

The  tool  to  help  determine  a  system’s  State  of  Resiliency  is  the  Resiliency 
Tier  Matrix  or  Framework,  with  varying  tiers  of  resiliency.  Before  proceed¬ 
ing  further,  an  explanation  of  a  Resiliency  Tier  Matrix  or  Framework  and 
how  it  is  built  is  appropriate.  Consider  the  spectrum  of  resiliency  divided 
into  five  states.  This  spectrum  ranges  from  the  worst  state  of  resiliency- 
exposed— through  the  states  of  confused,  aware,  and  operational  to  the  best 
state:  capable  (Table  1).  Appendix  B,  Table  Bl,  presents  further  descriptions 
equating  these  states  to  mission  accomplishment  and  operations. 


TABLE  1.  RESILIENCY  STATES  VS.  MISSION  AND  OPERATIONS 

Exposed 

No  mission 
accomplishment 

Ceases  to  function 

Confused 

Major  mission 
impairment 

Highly  impeded 

Aware 

Minimal  mission 

success 

Minimal  success 

Operational 

Effective  mission 
success  with  difficulties 

Effective 

Capable 

Mission  success  with  no 
difficulties 

Highly  effective 

The  question  now  arises:  How  is  a  system  placed  in  one  of  these  states? 
Measurable  criteria  (parameters,  techniques,  or  metrics)  help  in  con¬ 
structing  the  matrix.  The  key  criteria  are  those  that  help  define  this 
multidimensional  concept.  This  set  of  criteria  includes  system  charac¬ 
terization,  operator  confidence  in  the  system,  effectiveness  of  the  security 
precautions,  continuity  of  operations,  and  preparedness.  Appendix  B,  Table 
B2,  further  explains  these  criteria.  Each  of  these  can  further  be  subdivided 
depending  on  the  interest  and  the  importance  of  any  parameter  in  Figure  1, 
Resiliency  Umbrella.  The  matrix  begins  to  take  shape  in  Table  2. 
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The  intent  of  this  framework  is  to  produce  a  more  complete  picture  of  the 
system  and  the  forces  pulling  on  resiliency.  As  mentioned  earlier,  what  may 
be  important  one  day  may  not  be  important  the  next.  This  is  a  way  to  set  up 
a  score  card  and  evaluate  the  resiliency  of  a  system.  The  weighting  of  the 
criteria  would  be  set  according  to  the  priorities  of  those  criteria.  In  addition 
this  framework  also  provides  a  means  of  analyzing  vulnerabilities,  evalu¬ 
ating  tradespace,  and  comparing  various  courses  of  action.  Some  benefits 
(IBM,  2009,  p.  11)  for  constructing  such  a  framework  are: 

•  Aligning  capability  directly  to  mission; 

•  Projecting  potential  resiliency  investments; 

•  Improving  risk  mitigation  and  planning;  and 

•  Enhancing  preemptive  vs.  reactive  management. 

Some  key  challenges  (IBM,  2009,  p.  7)  for  constructing  such  a  framework  are: 

•  Viewing  resiliency  as  a  strategic  enabler.  Resiliency  has 
strategic  importance.  A  resiliency  strategy  would  be  a  single, 
integrated  plan  embraced  and  executed  by  all  parts  of  the 
organization.  It  would  focus  on  delivering  mission  capability.  It 
would  be  the  catalyst  to  higher  levels  of  performance.  Drawing 
together  the  different  components,  the  overall  result  would 
be  greater  than  the  parts  alone.  Senior  leadership  should  be 
committed  to  a  single  resiliency  strategy.  This  strategy  aligns 
with  organizational  goals  to  provide  a  holistic  approach  over 
mission-wide  systems  (McLaren,  2009). 

•  Defining  the  value  of  mission  resiliency.  “Mission  resil¬ 
iency  encompasses  a  proactive  approach  that  systematically 
prepares  for  potential  disruption  as  opposed  to  waiting  for  a 
disruptive  event  to  occur”  (Peake,  Underbrink,  &  Potter,  2012, 
p.  31).  Understanding  resiliency  in  the  mission  environment  is 
a  significant  step  in  system  development  and  security.  A  resil¬ 
ient  mission  system  is  more  capable  and  more  adaptable  than 
the  tools  used  against  it.  Its  value  is  in  less  complexity  and  cost 
of  securing  mission  systems.  “The  focus  on  mission  resilience 
extends  the  scope  of  past  security  practices  while  simultane¬ 
ously  honing  in  on  mission-critical  systems,  networks,  and 
processes”  (Peake  et  al.,  2012,  p.  29). 
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•  Working  with  advanced  technologies.  This  provides  the 
opportunity  to  assist  in  developing  and  integrating  state-of- 
the-art  solutions  to  meet  time-critical  needs.  As  an  added 
benefit,  it  provides  opportunities  for  proactive  and  independent 
research,  analysis,  testing,  and  prototype  development  to  mis¬ 
sion  requirements. 

•  Maintaining  continuous  availability  of  mission  systems. 

This  type  of  system  visibility  leads  to  assuring  uninterrupted 
availability  of  critical  mission  systems,  without  need  for 
failover  mechanisms  or  recovery  operations. 

•  Linking  capabilities  to  mission  requirements.  Building 
resilience  into  a  system  from  the  start  requires  an  understand¬ 
ing  of  the  mission,  the  environment,  and  potential  risks.  These 
systems  are  the  capabilities  that  satisfy  the  mission  require¬ 
ments.  Linking  the  capabilities  and  mission  requirements  and 
evaluating  their  effectiveness  in  a  hostile  environment  should 
be  done  early  in  the  life  cycle  of  a  program. 


Using  Resiliency  Tiers  in  Defining 
an  Architectural  Approach 

Resiliency  Tiers  define  levels  of  resiliency  to  match  mission  require¬ 
ments.  Resiliency  Tiers  span  all  domains,  services,  or  technologies  and 
provide  insight  for  building  a  resilient  architecture.  The  intent  is  that  this 
Resiliency  Tier  Framework  provides  an  objective  scale  for  the  classification 
of  mission  requirements.  This  scale  is  a  set  of  consistent  concepts,  mea¬ 
surements,  or  criteria  applied  to  mission  systems  or  capabilities.  This  set 
links  technical  resiliency  requirements  to  capabilities.  Mission  resiliency 
requires  an  architectural  approach  spanning  the  breadth  of  military  and 
government  capabilities.  Resiliency  Tiers  (IBM,  2009,  p.  10)  help  to  classify 
mission  requirements  by: 

•  Defining  a  broad  continuum  of  mission  resiliency  require¬ 
ments  that  apply  to  all  processes,  services,  development,  and 
missions; 

•  Linking  those  requirements  to  a  set  of  technology  criteria  that 
address  all  capabilities  and  resources  in  the  mission  environ¬ 
ment;  and 
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•  Providing  technical  characteristics,  criteria,  and  metrics  to 
measure  mission  resiliency  expectations,  and  to  monitor  and 
manage  ongoing  operations. 

This  process  develops  an  effective  holistic  Resiliency  Index.  The  whole  is 
greater  than  the  sum  of  each  functional  component’s  contribution.  This 
index  may  also  help  in  identifying  how  to  maximize  the  architecture  and 
optimize  investment. 


Mission  resiliency  requires  an  architectural 
approach  spanning  the  breadth  of  military  and 
government  capabilities. 


Benefits  of  Resiliency  Tiers 

Defining,  developing,  and  maintaining  Resiliency  Tiers  and  associated 
resilient  capabilities  have  a  number  of  benefits  (IBM,  2009,  p.  11),  such  as: 

•  Better  mission-to-technology  alignment; 

•  Clear  rationalization  of  investments  in  resilient  capabilities; 

•  Greater  opportunities  for  improvements  to  risk  planning, 
strategy,  and  architecture; 

•  More  prescriptive  management  of  the  mission  environment  to 
achieve  system-wide  resiliency; 

•  Assistance  in  gap  analysis  across  mission,  service,  and  tech¬ 
nology  domains; 

•  Help  in  bridging  the  communications  and  planning  gaps  for 
mission  continuity  resiliency  and  planning;  and 

•  Integration  of  mission  requirements  with  a  system-wide 
approach  to  achieve  greater  affordability. 

As  a  management  tool,  the  Resiliency  Tier  Framework  offers  a  way  to 
compare  various  programs,  systems,  and  capabilities  in  terms  of  potential 
tradespace,  cost  savings,  or  capability  optimization. 
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How  Resiliency  Tiers  Are  Used 

The  Resiliency  Tier  Framework  supports  every  aspect  of  the  mission 
system.  In  an  analysis,  this  framework  can  address  alignment  of  resiliency 
strategies  with  mission  needs,  can  guide  the  mitigation  of  adverse  actions, 
and  can  address  all  mission  and  system  components. 

These  tiers  are  able  to  help  conceptualize  and  align  mission  resiliency  needs 
in  multiple  scenarios.  Resiliency  Tiers  lead  to  a  comprehensive  picture  of 
systems  and  vulnerabilities,  and  eventually  an  understanding  of  specific 
levels  of  service.  Using  this  objective  and  quantitative  approach,  require¬ 
ments  definition  and  prioritization  ensure  that  the  resiliency  objectives  and 
acceptable  costs  are  integral  to  the  overall  mission  capability. 

An  organization  can  also  use  Resiliency  Tiers  for  guidance  to  mitigate  the 
potential  or  existing  chaos  caused  by  external  forces.  These  tiers  provide 
a  framework  for  understanding  the  overall  health  of  the  mission  area  and 
systems.  Similar  to  the  IBM  analysis.  Resiliency  Tiers  can  help  reconcile 
mission  resiliency  requirements  and  guide  the  infrastructure  require¬ 
ments,  architectural  design  decisions,  and  major  initiatives  that  will  be 
implemented  to  achieve  the  desired  future  resilient  environment  (IBM, 
2009,  p.  12). 

Lastly,  a  tiered  resiliency  approach  enables  the  warfighter  to  define  a  repli¬ 
cable  and  measurable  framework  that  can  address  all  mission  components 
including  weapon  systems,  force  capabilities,  and/or  government  actions 
(IBM,  2009,  p.  13).  It  can  provide  a  range  of  resiliency  requirements  as  well  as 
mitigating  actions.  In  addition,  the  tiered  resiliency  approach  may  also  apply 
to  a  wide  range  of  government  actions  and  resiliency  mitigations  such  as 
diplomacy,  technical  redundancy,  force  structures,  and  economic  measures. 

Five  Tiers  of  Resiliency 

This  framework  has  five  tiers  for  resiliency  estimation  (Table  2).  Each 
tier  serves  as  a  set  of  guidelines  that  specifies  the  characteristics  commen¬ 
surate  with  each  tier  condition  for  each  of  five  criteria:  System,  Confidence, 
Security,  Continuity  of  Operations,  and  Preparedness.  These  criteria  span 
the  five  Resiliency  Tiers  (defined  as  Capable  [V],  Operational  [IV],  Aware 
[III],  Confusion  [II],  and  Exposed  [I]).  When  taken  as  a  range,  the  Resiliency 
Tiers  translate  into  a  conceptual  view  of  the  resiliency  status  of  the  overall 
mission  system. 
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The  criteria  may  be  any  number  of  parameters  or  techniques,  which  are 
important  at  the  time.  Table  3  is  a  representative  example  of  a  populated 
Table.  (Appendix  B,  Table  B3  has  more  details  in  developing  this  matrix.)  For 
instance,  Preparedness  is  one  of  those  criteria.  The  Capable  Resiliency  Tier 
defines  Preparedness  as  having  a  holistic  approach  to  resiliency;  whereas  the 
Operational  Resiliency  Tier  classifies  this  as  having  specific  plans  in  place 
to  address  resiliency.  Depending  on  the  mission  resiliency  requirements, 
either  level  might  provide  adequate  preparedness;  however,  the  Capable 
Resiliency  Tier  provides  a  complete  strategy  for  addressing  resiliency.  The 
holistic  strategy  for  the  Capable  Resiliency  Tier  reduces  the  effects  of  out¬ 
side  forces  to  planned  courses  of  action  and  continuous  vigilance,  whereas 
the  Exposed  Resiliency  Tier  provides  no  indication  of  preparedness  for  a 
hostile  environment.  Again,  depending  on  mission  requirements,  any  level 
may  provide  adequate  resiliency;  however,  the  Capable  Resiliency  Tier 
provides  for  the  most  complete  level  of  Preparedness  for  mission-critical 
functions.  A  similar  analysis  is  possible  with  each  Criteria  or  row. 

The  outcome  of  this  assessment  defines  a  set  of  immediate  actions  to 
improve  the  resiliency  of  mission  systems.  Some  actions  would  result  in 
the  development  of  a  longer  term,  strategic  roadmap  of  major  initiatives  that 
would  help  meet  mission  expectations  for  future  applications. 


Guidance  on  Scoring 

When  undertaking  a  resiliency  assessment,  the  "how  good"  or  "how 
bad"  analysis  addresses  each  criteria  individually  (National  Patient 
Safety  Agency,  2008,  p.  14).  This  is  a  consequence  of  the  mission 
environment.  Consequence,  in  this  context,  means  the  condition 
or  outcome  of  a  mission  capability  in  reaction  to  an  outside  force 
(National  Patient  Safety  Agency,  2008,  p.  4).  Clearly,  there 
maybe  more  than  one  consequence  for  a  single  capability. 


Defense ARJ,  July  2015,  Vol.  22 No.  3:294-324 


307 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


TABLE  3.  TABLE  OF  RESILIENCY  TIERS 

^'''~\Tiers 

Criteria 

Priority 

Weighting 

V  IV 

[Capable]  [Operational] 

Scale 

7  2 

SYSTEM 

Overview 

Highly  capable  Effective 

Normal 

Operations 

Full  capabilities  Maintains  normal 

on-line  operations,  reached 

new  equilibrium 

Protection 

Protected  Protection  measures 

in  place 

Corrective 

Actions 

Cohesive  actions  Synergy  of  actions 

among  all  players  among  most  actors 

Vulnerabilities 

Potential  vulnerabili-  Know  of  most 
ties  identified  vulnerabilities 

Planning 

Holistic  resilience  Resiliency  measures 

strategy 

Mitigations 

Attacks  have  little  or  Successful  in 

no  effect  on  mitigating  or  avoiding 

operations  most  attacks 

Vigilance 

Method  to  identify  Addresses  obvious 

new  vulnerabilities  vulnerabilities 

Confidence 

High  Moderate 

Security 

High  Effective 

Continuity  of 
Operations 

Maximum  Able  to  operate 

effectively 

Preparedness 

Holistic  strategy  Specific  plans  in  place 

approach 

BEST 
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TABLE  3.  TABLE  OF  RESILIENCY  TIERS 

,  CONTINUED  | 

III 

II 

1 

[Aware] 

[Confusion] 

[Exposed] 

J 

4 

5 

Minimum  mission 
accomplished 

Problems  meeting  any 
mission  needs 

Ineffective 

Struggles  to  stay  ahead 
of  problems 

Experiencing  outages, 
delays,  "blackouts," 
etc.— confused  with 
anomalies 

System  failure,  it 
crashes 

Some  protection 
available 

"Band-aid"  protection 

No  protection 

Collaboration  of  effort 
to  address  issues 

Attempting  to  resolve 
from  within— disjointed 
actions 

No  clue  what  to  do 

Vulnerabilities  exist 

Few  vulnerabilities 
known 

Unaware  of 
vulnerabilities 

Realistic  impact 
assessment 

Minimal  resiliency 
actions  available 

No  resiliency  designed 
in  system 

Some  proactive 
measures  in  place 

Reactive  measures 
taken 

No  measures  available 

Aware  of  attacks 

Can  spell  resiliency 

Clueless 

Medium 

Low 

Nonexistent 

Appears  to  be 
adequate 

Minimal  with  breaches 

None 

Barely  meeting 
requirements 

Failing 

Complete  breakdown 

Minimal  to  acceptable 

Insufficient 

None 

WORST 
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Qualitative  and  quantitative  techniques  assess  and  score  the  consequences. 
Wherever  possible,  consequences  should  use  objective  definitions  across  dif¬ 
ferent  criteria  within  each  tier  to  ensure  consistency  in  the  process.  Despite 
defining  each  condition  as  objectively  as  possible,  scoring  the  consequences 
will  inevitably  involve  a  degree  of  subjectivity.  Figure  2  contains  the  flow 
diagram  for  the  Resiliency  Tier  assessment. 


FIGURE  2.  RESILIENCY  TIER  ASSESSMENT  FLOW  CHART 
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•  Compare  Analysis  of  Alternatives 
Solutions 

•  Compare  with  Another  System  Rl 

•  Deployment 

•  Future  Actions/Improvements 

•  Improve  Resiliency 

•  Investment  Options 

•  Research  and  Development 

•  Strategic  Roadmap 

•  Sustainment 

•  Etc. 


Table  3,  Table  of  Resiliency  Tiers,  provides  the  framework  to  obtain  an 
assessment  of  the  State  of  Resiliency  of  a  specific  mission  system.  The 
process  is: 

•  Select  the  mission  system  to  review. 

•  Define  explicitly  the  conditions  (internally  or  externally)  of  the 
adverse  consequences  that  are  either  encountered  or  might  be 
encountered. 
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•  Go  to  each  row  (criteria)  in  the  table  and  identify  the  appropri¬ 
ate  description,  or  tier,  under  the  adverse  condition.  Appendix  B 
contains  further  details  for  each  term  and  description.  Record 
the  scale  number  at  the  top  of  each  column.  If  a  weighted  value 
exists,  multiply  the  scale  number  by  the  weighted  value. 

•  Once  all  12  rows  are  characterized,  add  all  the  scores  based  on 
the  scale  value  (with  or  without  weighted  values)  for  each  row. 

The  total  is  the  Resiliency  Index. 

•  A  variation  to  this  table  would  be  to  change  to  another  or  dif¬ 
ferent  set  of  criteria  or  parameters.  Add  or  delete  a  row.  If  one 
is  added,  establish  the  corresponding  tier  structure  based  on 
the  new  criteria.  Keep  modifications  to  a  minimum.  One  of  the 
benefits  to  having  a  set  of  criteria  is  the  aspect  of  consistency 
in  application. 

This  provides  an  overall  resiliency  assessment  of  the  system:  the  greater 
the  score,  the  lower  the  resiliency.  The  scores  for  this  Resiliency  Tier 
Framework  (no  weighting)  would  range  from  12  (the  best)  to  60  (the  worst). 
Putting  these  scores  into  perspective,  compare  them  to  the  Chairman  of  the 
Joint  Chiefs  of  Staff  (CJCS)  risk  scale  as  part  of  the  C  JCS  Resiliency  Risk 
Spectrum  (Figure  3). 


FIGURE  3.  RESILIENCY  RISK  SPECTRUM 


CJCS  Risk  Ratings 

LOW  MODERATE  SIGNIFICANT  HIGH 


12  22  36  50  60 

Relative  Scale  for  Resiliency  Characterization 


The  following  is  an  example  of  how  this  Resiliency  Tier  Matrix  is  applied  to 
a  specific  situation  and  system.  Assume  a  large  satellite  terminal  is  located 
on  foreign  soil.  The  Status  of  Forces  Agreement  states  physical  protection  is 
the  responsibility  of  the  host  nation.  Further,  this  terminal  is  vintage  equip¬ 
ment  nearing  end  of  life.  A  local  protest  breaks  out  and  the  satellite  signal 
is  lost  for  the  first  time.  After  working  with  higher  headquarters  and  taking 
approved  mitigating  actions,  the  maintenance  crew  restores  the  system  to 
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full  operational  status  within  appropriate  restoration  time  frames.  Once  all 
activities  return  to  normal,  the  resiliency  assessment  (Figure  4)  uses  Table 
3,  highlighting  the  applicable  tiers  for  each  criteria  within  the  Resiliency 
Tier  Framework.  Refer  to  Table  3  for  the  cell  descriptions. 

The  sum  of  the  respective  scale  numbers  is  37.  This  number  is  displayed 
above  the  scale  in  Figure  5.  An  interpretation  of  this  State  of  Resiliency 
would  indicate: 

•  Increased  system  protection  is  imperative. 

•  Better  planning  for  such  events  is  necessary. 

•  Known  vulnerabilities  need  more  attention. 

•  The  system  is  getting  old. 


FIGURE  5.  RESILIENCY  RISK  SPECTRUM-STATE  OF  RESILIENCE 


37 


12  22  36  50  60 

Relative  Scale  for  Resiliency  Characterization 


These  four  items  would  lead  to  a  cost  analysis  of  whether  to  upgrade  or 
replace  the  system.  They  may  also  lead  to  a  political  discussion  on  the  Status 
of  Forces  Agreement  or  whether  or  not  the  site  should  remain  in  its  current 
location.  Looking  at  a  variation  of  the  situation  above  where  the  terminal 
never  goes  down,  discussions  would  be  much  different.  Many  of  the  cell 
evaluations  in  Figure  4  would  move  to  the  left. 

This  is  a  single  application  for  illustration  purposes;  however,  other  options 
could  be  to  maximize  architectural  designs,  optimize  investments,  and  dif¬ 
ferentiate  resiliency  between  two  systems  supporting  the  same  mission  or 
among  analysis  of  alternatives  solutions.  The  analysis  can  be  as  rigorous  as 
necessary  with  all  details,  a  subset  of  details,  or  limited  details  depending 
on  the  purpose  and  desired  outcome. 
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Summary 

The  tiered  approach  to  resiliency  can  aid  in  planning  for  adverse  or 
intrusive  events  proactively.  This  helps  maximize  return  on  investment 
from  assets,  technology,  and  people  at  the  time  when  needed  most.  Using 
Resiliency  Tiers  to  develop  effective  long-term  strategies  ensures  that 
shorter  term  tactical  actions  are  properly  aligned  and  supports  a  military 
capability  progress  along  the  resiliency  maturity  continuum.  Investing  in 
resiliency  measures  at  the  program  start  will  help  make  sure  that  long-term 
resiliency  investments  preserve  value  over  time. 
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Appendix  A 

Resiliency  Black  Box 

In  viewing  the  various  parameters  of  Figure  1,  Resiliency  Umbrella, 
resiliency  as  a  concept  has  many  moving  parts,  elements,  and  metrics  or 
components.  At  any  one  time,  any  of  these  can  be  a  driving  force  for  change. 
The  result  of  that  change  could  be  a  new  equilibrium  of  interaction  and 
collaboration.  One  way  to  visualize  this  interaction  is  to  see  resiliency  as 
a  black  box.  It  has  inputs  (data,  resources,  and  feedback)  and  has  an  out¬ 
put.  In  a  more  strict  sense,  a  “black  box”  analysis  “of  [a]  system  contains 
formulas  and  calculations  that  the  user  does  not  see  ...  to  use  the  system. 
Black  box  systems  are  often  used  to  determine  optimal  trading  practices 
[in  investments]”  (Black  Box  Model,  n.d.).  In  this  case,  the  Resiliency  Black 
Box  Model  depicted  in  Figure  A-l  illustrates  how  the  various  inputs— 
Adjustments,  Mitigation  Actions,  and  As  Designed  or  Modified  (internally) 
and  Environment  (externally)— when  altered,  can  reach  a  new  system 
equilibrium  or  resiliency  state.  Putting  it  another  way,  equilibrium ...  refers 
to  a  steady  status  in  which  model  state  variables  reach  a  dynamical  bal¬ 
ance  (Wang,  2009,  p.  9).  This  dynamic  balance  could  result  in  a  system 
achieving  a  reasonable,  acceptable,  or  tolerable  resiliency  state.  All  the 
parameters  contribute  to  the  system  equilibrium,  whether  new  or  a  return 
to  the  previous  state.  The  mission  planner  must  assess  the  new  resiliency 
state.  If  the  resiliency  state  is  unacceptable,  a  resiliency  analysis  needs  to 
be  accomplished  to  determine  the  best  course  of  action  that  has  a  holistic 
effect  on  the  system. 


FIGURE  A1.  RESILIENCY  "BLACK  BOX"  MODEL  DIAGRAM 
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Generally,  systems  operate  under  two  states:  benign  and  hostile.  The  evalu¬ 
ation  of  these  states  occurs  in  the  “Situation  Assessment”  block.  Use  the 
parameters,  conditions,  and/or  metrics  from  Figure  1  to  define  and  evaluate 
effectiveness.  Pulling  all  of  these  together  helps  develop  a  Resiliency  Index. 


TABLE  A2.  DESCRIPTION  OF  THE  DOMAINS 


Criteria 

Description 

Scale 

The  measure  of  “how  good”  or  “how  bad”  a  system  is 
relative  to  the  Resiliency  Tiers. 

System 

A  functionally,  physically,  and/or  behaviorally  related 
group  of  regularly  interacting  or  interdependent  elements. 
(Joints  Chiefs  of  Staff,  2011,  p.  GL-17) 

Confidence 

The  feeling  or  belief  that  one  can  rely  on  someone 
or  something;  firm  trust.  (Oxford  Dictionary,  online 
reference) 

Security 

Measures  taken  by  a  military  unit,  activity,  or  installation 
to  protect  itself  against  all  acts  designed  to,  or  which  may, 
impair  its  effectiveness.  (JP  1-02,  page  226,  8  November 
2010). 

Continuity  of 
Operations 

The  degree  or  state  of  being  continuous  in  the  conduct 
of  functions,  tasks,  or  duties  necessary  to  accomplish 
a  military  action  or  mission  in  carrying  out  the  national 
military  strategy.  (Joint  Chiefs  of  Staff,  2010,  p.  54) 

Preparedness 

A  state  of  readiness,  especially  for  war.  (Oxford  Dictionary, 
online  reference) 
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TABLE  A3.  DESCRIPTION  OF  TABLE  ELEMENTS  1 

Criteria 

Tier 

Tier  Description 

Tier  Explanation 

Scale 

1-5 

This  is  an  attempt  to  quantify  the 
current  condition  of  a  system  or 
capability.  The  lower  the  score  the  more 
resilient  a  system  or  capability  is. 

V 

Highly  capable 

System  is  highly  capable  of  completing 
the  mission. 

IV 

Effective 

System  experiences  some  minor 
problems  but  effectively  accomplishes 
the  mission. 

£ 

o 

> 

(1) 

III 

Minimum 

mission 

accomplished 

System  is  struggling  to  meet  mission 
minimum  requirements. 

> 

O 

II 

Problems 
meeting  any 
mission  needs 

System  can’t  meet  most  mission 
requirements,  is  distracted  by 
problems,  and  cannot  keep  up  with 
mitigating  actions. 

1 

Ineffective 

System  cannot  meet  mission 
requirements.  Problems  have  the 
system  on  the  verge  of  collapsing. 

SYSTEM 

V 

Full  capabilities 
on-line 

System  is  running  all  subsystems, 
processes  and  applications  with  no 
problems. 

</> 

C 

o 

IV 

Maintains  normal 
operations, 
reaches  new 
equilibrium 

System  is  running  normal  operations; 
however,  it  is  continuously  adjusting  for 
disruptions.  Each  adjustment  allows  the 
system  to  reach  a  new  equilibrium  of 
operations. 

2 

a> 

a 

O 

15 

E 

III 

Struggles  to 
stay  ahead  of 
problems 

System  cannot  maintain  mission 
accomplishment.  It  is  struggling  to  stay 
ahead  of  the  disruptions.  Subsystems, 
processes,  and  applications  are  failing. 

o 

z 

II 

Experiencing 
outages,  delays, 
"blackouts,"  etc. 
—confused  with 
anomalies 

System  is  spending  more  time 
addressing  disruptions  than 
accomplishing  the  mission.  The 
outages,  delays,  and  disruptions  are  a 
distraction  to  the  mission.  Anomalies 
present  no  easy  problems. 

1 

System  failure,  it 
crashes 

System  crashes  or  is  near  to  crashing 
under  the  weight  of  disruptions. 
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TABLE  A3.  DESCRIPTION  OF  TABLE  ELEMENTS,  CONTINUED 

Criteria 

Tier 

Tier  Description 

Tier  Explanation 

V 

Protected 

System-wide  protection  has  proactive 
processes  in  identifying  and  mitigating 
disruptions.  System  is  alert  to  new 
disruptions  and  puts  corrective 
measures  in  place  immediately. 

c 

o 

'5 

IV 

Protection 
measures  in 
place 

System  has  many  protective  measures 
in  place.  It  is  not  totally  proactive  in 
its  corrective  action.  However,  it  is 
able  to  identify  problems  and  react 
appropriately  and  swiftly. 

Q) 

o 

k. 

Q. 

III 

Some  protection 
available 

System  has  elementary  protection 
measures.  Primary  mode  of  correction 
is  reactionary  to  disruptions.  Little  time 
is  available  to  be  proactive. 

II 

‘Band-Aid’ 

protection 

No  system-wide  protection  in  place. 
Disruptions  circumvent  any  protection 
measures  attempted.  Fixes  turn  out  to 
be  band-aids  addressing  symptoms 
and  not  causes. 

1 

No  protection 

System  has  little  or  no  protection  at  all. 

SYSTEM 

V 

Cohesive  actions 
among  all 
players 

When  disruptions  occur,  there 
is  a  single  focused  team  across 
the  organization  addressing  any 
disruptions. 

(/> 

C 

o 

IV 

Synergy  of 
actions  among 
most  actors 

Pockets  of  excellence  pop  up 
throughout  the  organization  to  address 
any  disruptions.  There  is  a  coordinated 
synergy  among  all  actions  taken.  The 
effectiveness  of  these  actions  is  greater 
than  the  sum  of  the  individual  actions. 

< 

a> 

> 

'5 

u 

O 

a- 

o 

u 

III 

Collaboration  of 
effort  to  address 
issues 

There  is  a  collaborative  effort  to 
address  disruptions.  This  effort 
is  initiated  by  the  most  affected 
subsystem  or  process  or  application. 
Coordination  is  not  readily  obtained.  It 
takes  time  to  address  issues. 

II 

Attempting 
to  resolve 
from  within- 
disjointed 
actions 

Individual  offices  work  independent 
of  each  other  in  attempting  to 
solve  any  issues.  In  some  cases  it  is 
counterproductive. 

1 

No  clue  what  to 
do 

Little  or  no  effort  is  put  forward  to 
address  disruptions. 
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TABLE  A3.  DESCRIPTION  OF  TABLE  ELEMENTS,  CONTINUED 


Criteria 

Tier 

Tier  Description 

Tier  Explanation 

V 

Potential 

vulnerabilities 

identified 

System  is  aware  of  all  vulnerabilities, 
has  a  means  of  identifying  new 
vulnerabilities,  and  is  able  to  project 
vulnerabilities  that  result  from  new 
technology  development. 

t/> 

Q) 

■M 

IV 

Know  of  most 
vulnerabilities 

System  knows  of  its  primary 
vulnerabilities  and  can  sense  new 
vulnerabilities  as  they  manifest 
themselves.  System  has  an  excellent 
means  of  assessing  new  technologies 
for  possible  impacts. 

A 

2 

Q) 

C 

3 

> 

III 

Vulnerabilities 

exist 

System  knows  vulnerabilities  exist; 
however,  it  is  not  aware  of  most  of 
them.  It  reacts  to  disruptions.  Has  no 
ability  to  project  vulnerabilities  from 
new  technology. 

Z 

II 

Few 

vulnerabilities 

known 

System  has  the  basic  understanding 
of  vulnerabilities  and  is  aware  of  most. 

Has  no  effort  in  place  to  address  new 
vulnerabilities  ahead  of  disruptions. 

LU 

h 

(/) 

>- 

t/> 

1 

Unaware  of 
vulnerabilities 

System’s  awareness  of  vulnerabilities  is 
no  more  than  elementary  and  probably 
much  less. 

Planning 

V 

Holistic 

resilience 

strategy 

System  has  a  resilience  strategy  or 

Plan  in  place  that  is  supported  by  the 
entire  organization.  It  is  ingrained  in 
the  architecture  of  the  system  and 
culture  of  the  organization.  It  covers 
current  conditions  and  future  projected 
environments.  It  has  provisions  for 
training  and  education. 

IV 

Resiliency 

measures 

System  has  a  coherent  set  of  resiliency 
measures  that  apply  to  any  and  every 
subsystem,  capability  or  process.  The 
concept  is  accepted  organization 
wide;  however,  emphasis  is  different  in 
different  work  centers  or  offices. 

III 

Realistic  impact 
assessment 

Realistic  risk  and  operational 
assessments  provide  focused  courses 
of  action  and  necessary  organizational 
involvement  for  current  conditions.  No 
long-term  plan. 
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TABLE  A3.  DESCRIPTION  OF  TABLE  ELEMENTS,  CONTINUED 


Criteria  Tier  Tier  Description  Tier  Explanation 

Minimal  Any  resiliency  actions  available  are 

resiliency  reactive  and  localized  to  specific 

actions  subsystems,  capabilities  or  processes, 

available  There  is  no  effort  to  address  issues  at  a 

system  level. 

No  resiliency  Resiliency  is  taken  for  granted.  There 
designed  in  is  no  underlying  theme  or  approach  to 

system  Resiliency. 


V 

Attacks  have 
little  or  no 
effect  on 
operations 

Attacks  are  generally  insignificant. 

System  is  able  to  tolerate  and  mitigate 
them  and  continue  operations  as 
normal. 

c 

IV 

Successful  in 
mitigating  or 
avoiding  most 
attacks 

Attacks  are  annoying.  Specific  actions 
need  to  be  taken;  however,  they  are 
successful  in  mitigating  any  effects. 

Mitigatio 

III 

Some  proactive 
measures  in 
place 

Attacks  are  serious  and  cannot 
be  ignored.  More  reactive  than 
proactive  measures  are  necessary. 

Many  consequences  of  attacks  are 
unexpected. 

II 

Reactive 
measures  taken 

Attacks  are  critical  to  the  system 
operation  and  mission  accomplishment. 
The  reactive  measures  do  not  handle  all 
of  the  attacks. 

1 

No  measures 
available 

Attacks  are  catastrophic  and  result  in 
system  shutdown. 

Vigilance 

V 

Method  to 
identify  new 
vulnerabilities 

System  has  means  to  research  and 
assess  new  sources  of  disruptions 
and  the  vulnerabilities.  It  is  generally 
expected  that  the  system  is  prepared 
for  new  technology  attacks. 

IV 

Addresses 

obvious 

vulnerabilities 

System  is  in  place  to  address  all  known 
vulnerabilities.  The  ability  to  address 
the  surfacing  of  new  vulnerabilities  is  a 
reactive,  but  effective,  process. 

III 

Aware  of 
attacks 

System  is  aware  of  new  vulnerabilities 
as  they  are  attacked.  It  has  no  means  of 
identifying  the  new  vulnerabilities  prior 
to  an  attack. 
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TABLE  A3.  DESCRIPTION  OF  TABLE  ELEMENTS,  CONTINUED  j 

Criteria 

Tier 

Tier  Description 

Tier  Explanation 

Z 

ill 

H 

■o 

o 

3 

C 

5 
c 

0 

U 

6 

II 

Can  spell 
resiliency 
[surprised  by 
attacks] 

System  needs  to  take  time  to  study 
an  attack  and  the  symptoms  before 
it  can  generate  the  awareness  of 
a  new  vulnerability.  It  may  not  be 
able  to  correct  or  mitigate  the  new 
vulnerability. 

W 

u 

c 

ro 

'5 

> 

1 

Clueless  [does 
not  know  what 
to  do] 

System  seeks  outside  help  because 
it  does  not  understand  the  new 
vulnerability  or  the  extent  it  affects  the 
mission. 

V 

High 

System  confidence  is  high,  fully 
confident  that  the  system  or  capability 
will  perform  the  mission  with  little  or  no 
disruptions  affecting  operations. 

o 

o 

c 

IV 

Moderate 

System  has  moderate  confidence  that 
it  will  accomplish  the  mission  in  spite  of 
potential  disruptions. 

Q) 

5 

sc 

c 

0 

u 

III 

Medium 

Medium  confidence  illustrates  concern 
over  mission  accomplishment  and 
integrity  of  the  system. 

II 

Low 

Low  confidence  lacks  any  belief  that 
the  system  can  be  counted  on  to  do  the 
mission. 

1 

Nonexistent 

No  confidence  means  that  the  system  is 
not  acceptable. 

V 

High 

There  are  no  acts  that  can  bypass  or 
contravene  security  policies,  practices, 
or  procedures. 

IV 

Effective 

In  an  environment  of  minor  security 
breaches,  security  policies,  practices, 
or  procedures  are  able  to  protect 
the  system  effectively  for  mission 
accomplishment. 

Security 

III 

Appears  to  be 
adequate 

On  the  surface,  security  policies, 
practices,  or  procedures  appear  to  be 
effective;  however,  security  problems 
exist  and  often  prevail. 

II 

Minimal  with 
breaches 

Security  breaches  dominate  the  system 
and  create  an  environment  of  mistrust. 
This  leads  to  minimal  to  no  mission 
accomplishment 

1 

None 

There  are  no  security  policies,  practices, 
or  procedures  in  place  to  prevent 
breaches. 
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Appendix  B 

Resiliency  Tier  Descriptions 


TABLE  Bl.  DESCRIPTION  OF  THE  FIVE  TIERS 


Tier 

Description 

V  Fully  Capable 

May  result  in  a  slight  perturbation  in  operations; 
however,  the  system/capability  continues  operating 
with  nothing  more  than  a  “hiccup.”  Any  disruption 
is  an  exceptional  circumstance.  (Insignificant 
disruptions) 

VI  Operational 

May  experience  a  disruption  resulting  in  possible 
resets  or  reboots;  however,  mission  is  accomplished 
and  the  disruptions  are  immediately  isolated  and 
mitigated.  Disruptions  can  occur  at  any  time; 
however,  they  are  not  showstoppers.  (Negligible 
disruptions) 

III  Aware 

Is  cognizant  of  operating  environment,  hazards 
therein,  and  vulnerabilities.  Disruptions  have  a 
reasonable  likelihood  of  occurring  at  any  time. 
Mitigating  actions  are  not  always  effective.  Capability 
tolerates  disruptions,  but  also  does  not  handle  the 
consequences  well.  (Moderate  disruptions) 

II  Confusion 

Disruptions  result  in  permanent  partial  disability 
or  operational  incapacity.  Likelihood  of  disruptions 
happening  is  high.  There  is  no  requisite 
understanding  of  the  problems.  (Extensive 
disruptions) 

1  Exposed 

Disruptions  are  inevitable  and  greatly  impact  the 
system/capability.  The  capability  is  unprotected, 
totally  exposed  to  hazardous  environment.  Damage 
may  be  irreversible.  (Catastrophic  disruptions) 
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A  prime  managerial  concern  is  howto  decide  which  investment  alternatives 
provide  the  greatest  return  with  least  risk  of  loss.  In  civilian  organizations, 
numerous  methods  and  formulas  assist  these  decisions.  However,  in  military 
and  other  governmental  agencies,  these  methods  often  fall  short  because  typical 
governmental  investments  do  not  have  a  monetary  return.  The  processes 
underpinning  governmental  resource  allocation  and 
acquisition  decisions  are  often  cumbersome  and 
time  consuming.  In  this  article,  the  authors  present 
a  unique  application  of  composite  indexing  methods 
to  compare  the  return  on  investment  in  military  equipment.  They 
posit  that  this  analytical  method  can  improve  government  agencies’  invest¬ 
ment  decisions  for  capital  equipment,  especially  when  methods  that  are  more 
laborious  cannot  be  executed  in  the  allotted  time  frame. 
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A  prime  managerial  concern  is  how  to  select,  among  a  range  of  invest¬ 
ment  alternatives,  the  option  that  provides  the  greatest  return  with  least 
risk  of  loss.  In  civilian  organizations,  numerous  methods  and  formulas  such 
as  Net  Present  Value,  Return  on  Investment  (ROI),  and  Return  on  Assets 
address  these  issues  (Brealy,  Myers,  &  Allen,  2011).  However,  in  military 
and  other  governmental  agencies,  these  methods  often  fall  short  because 
government  investments  do  not  offer  a  monetary  return.  Rather,  they  pro¬ 
vide  intangible  returns  such  as  national  defense,  public  safety,  goodwill,  and 
other  public  goods  that  are  difficult,  but  not  impossible,  to  quantify  (Oswalt 
et  al.,  2011).  As  Gonzalez,  Perera,  and  Correa  (2003)  noted,  "the  economic 
valuation  of  nonmarket  goods. ..is  aimed  at  obtaining  a  monetary  assess¬ 
ment  of  the  welfare  or  utility  gain  (or  loss)  experienced  by  a  certain  group  of 
people  from  the  improvement  of  (or  damage  to)  a  nonfinancial  asset"  (p.  65). 

Numerous  economic  models  for  calculating  ROI  exist,  and  most  require  only 
a  few  basic  inputs  such  as  costs,  benefits,  time  horizon,  and  risks  (Bailey, 
2015).  The  benefit  of  calculating  ROI  of  government  investments  is  to  save 
costs  over  other  alternatives  (Bailey,  Mazzuchi,  Sarkani,  &  Rico,  2014),  but 
scholarly  research  into  assessing  the  ROI  of  complete  military  systems  is 
lacking.  In  this  article,  we  present  a  method  that  efficiently  compares  equip¬ 
ment  options  using  a  composite  index  that  generates  a  normalized  measure 
of  performance  return.  By  objectively  assessing  equipment’s  ROI,  leaders 
can  eliminate  low-value  and  inefficient  programs,  ultimately  saving  U.S. 
taxpayer  dollars. 


Background  and  Literature  Review 

Department  of  Defense  (DoD) 
budget  and  acquisition  decisions 
are  lengthy  processes  governed  by 
hundreds  of  federal  laws  and  prac¬ 
tices  (Chairman  of  the  Joint  Chiefs  of  Staff, 

2015),  often  producing  suboptimal  and  ineffective  results 
(Government  Accountability  Office,  2008). 

Such  decisions  involve  professionals  from 
many  government  entities  and  disciplines, 
as  well  as  politicians  who  all  have  differ¬ 
ent  perspectives  on  the  best  way  to  invest 
scarce  public  funds.  As  decision  analysts  at 
Headquarters  Marine  Corps,  we  saw  lead¬ 
ership  request  analytical  support  to  make 
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considerable  performance,  capacity,  and  resource  trade-offs  quickly  during 
all  phases  of  the  Planning,  Programming,  Budgeting,  and  Execution  system 
cycle.  Often,  these  decisions  are  made  with  an  incomplete  understanding 
of  an  investment’s  value  (return)  because  it  cannot  be  objectively  quanti¬ 
fied.  To  support  resource  allocation  decisions,  our  mission  was  to  provide 
accurate  and  timely  analyses  with  readily  available  information. 

In  fiscal  year  2014,  the  Marine  Corps  evaluated  its  strategic  equipment 
investment  initiatives  for  the  ground  combat  and  tactical  vehicle  (GCTV) 
fleet.  Between  2025  and  2035,  85  percent  of  currently  fielded  platforms 
within  the  GCTV  portfolio  are  projected  to  reach  the  end  of  their  service 
life,  necessitating  a  large  influx  of  capital  to  replace  or  sustain  GCTV  capa¬ 
bilities  (Dinwoodie,  2012).  In  addition,  all  these  investments  are  competing 
for  dwindling  funds  within  the  larger  Marine  Corps  budget  due  to  the 
2011  Budget  Control  Act  and  predicted  reductions  in  defense  spending 
(Krepinevich,  2012;  Liebman,  2013).  Inevitably,  declining  budgets  force 
trade-offs  among  important  projects. 


When  we  analyzed  GCTV  asset  options,  it  was  difficult  to  compare  the  costs 
and  returns  of  different  types  of  equipment  as  complete  systems.  As  Oswalt 
et  al.  (2011)  asserted,  “a  practice  or  methodology  does  not  exist  in  the  DoD 
to  capture  and  characterize  the  future  and  extended  value  accruing  to  users 
beyond  the  primary  recipients  of  the  investment”  (p.  126).  Boiling  complex 
military  systems  down  to  one  metric  is  difficult  for  three  main  reasons:  (a) 
vehicle  performance  measures  typically  cannot  be  aggregated  into  a  single 
overall  measure;  (b)  opinions  about  military  equipment’s  utility  differ  and 
are  often  subjective;  and  (c)  accepted  quantitative  methods  for  assessing 
overall  value  are  time-  and  resource-intensive. 


First,  performance  data  on  vehicles  are  typically 
measured  and  quantified  in  different  units  of  measure  for 
specific  characteristics  such  as  fuel  consumption 
in  miles  per  gallon;  payload  in  pounds;  and  speed 
in  miles  per  hour.  Within  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS), 
developing  achievable  requirements,  called 
Key  Performance  Parameters  (KPP)1  and  Key 
System  Attributes  (KSA),2  requires  establishing 
Measures  of  Performance  (MOP)  and  Measures 
of  Effectiveness  (MOE).  MOPs  are  “system-par¬ 
ticular  performance  parameters  such  as 
speed,  payload,  range. ..or  other  distinctly 
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quantifiable  performance  features"  (Defense  Acquisition  University,  n.d., 
para.  1).  MOEs  measure  operational  capabilities  in  terms  of  engagement 
or  battle  outcomes  (Department  of  the  Air  Force,  1996).  Elaborate  opera¬ 
tional  testing  and  evaluation  events  are  created  to  evaluate  these  measures 
(Gentner,  Best,  &  Cunningham,  n.d.).  Extensive  modeling  and  simulation 
events  evaluate  system  performance  in  scenarios  and  vignettes  (Gentner 
et  al.,  n.d.;  Lai  &  Lamoureux,  2012;  Lingel  et  al.,  2012).  While  these  methods 
assess  performance  and  provide  inputs  to  decisions,  they  are  not  structured 
to  create  a  singular  and  objective  measure  of  ROI.  Performance  metrics 
such  as  MOPs,  MOEs,  KPPs,  and  KSAs  can  be  compared  across  systems, 
but  cannot  be  aggregated  into  a  single  measure  of  overall  performance 
without  normalization. 


Second,  qualitative  value  assess¬ 
ments  of  military  equipment  are 
often  subject  to  biases  "because 
personal  cognitive  processes 
inform  how  individuals  understand 
their  environment"  (Reynolds, 
2015).  Consequently,  military  per¬ 
sonnel  have  different  qualitative 
biases  towards  equipment  based  on 
their  specific  experiences  (Simon, 
2004).  Conversely,  the  assessment 
of  a  financial  investment's  ROI  is 
simpler:  PrW 

Cost 

(Stickney,  Weil,  Schipper,  & 
Francis,  2010)  and  normalized  in  a 
common  measure:  dollars. 

Third,  a  significant  body  of  research 
and  accepted  practices  exists  to 
quantitatively  assess  qualita¬ 
tive  value  preferences,  but  these 
methods  are  time-  and  resource¬ 
intensive.  The  Analytic  Hierarchy 
Process  (AHP)  is  a  multilevel,  deci¬ 
sion-making  framework  that  allows 
“practitioners  to  assign  numerical 
values  to  what  are  abstract  concepts 
and  then  deduce  from  these  values 
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decisions  to  apply  to  a  global  framework”  (Saaty,  1988,  p.  110).  This  frame¬ 
work  allows  the  judgments  of  qualified  individuals  to  be  aggregated  into  a 
group  judgment.  Based  on  the  intensities  of  those  judgments,  an  output  with 
explicit  rules  for  allocating  resources  among  competing  projects  is  derived 
(Saaty,  2013). 

Similarly,  Multiple-Objective  Decision  Analysis  (MODA)  and  Value- 
Focused  Thinking  are  interrelated  methodologies  that  can  derive  value 
functions  that  map  performance  scores  to  value  metrics  (Parnell,  Bresnick, 
Tani,  &  Johnson,  2013).  MODA  “quantitatively  assesses  the  trade-offs 
between  conflicting  objectives  by  evaluating  an  alternative’s  contribution 
to  the  value  measure  and  the  importance  of  each  value  measure”  (Parnell 
et  al.,  2013,  p.  196).  However,  both  AHP  and  MODA  can  be  time  consuming 
and  difficult  to  execute  since  they  typically  require  significant  amounts  of 
senior  leadership  attention  (Triantaphyllou  &  Mann,  1995). 

Given  the  shortcomings  of  more  elaborate  methods,  our  desire  was  to  cre¬ 
ate  a  single  ROI  metric  that  provides  a  straightforward,  quantitative,  and 
objective  evaluation  of  options.  Our  method  utilizes  a  composite  index  to 
normalize  different  measures  and  aggregate  them  into  a  single  metric  facili¬ 
tating  holistic  comparison  of  multiple  system  alternatives.  In  this  case,  we 
used  established  KPPs  and  KSAs,  or  Performance  Metrics,  as  a  baseline.  We 
then  calculated  the  relative  deviation  of  multiple  platforms'  performance 
from  this  baseline.  This  method,  called  the  Distance  to  Reference  (DTR) 
technique,  “measures  the  relative  position  of  a  given  indicator  vis-a-vis  a  ref¬ 
erence  point”  (Organisation  for  Economic  Co-Operation  and  Development, 
2008,  p.  28).  The  disparate  measures  are  normalized  by  dividing  the  tested 
performance  value’s  distance  from  the  reference  point  by  the  reference 
point.  Once  all  metrics  are  converted,  they  are  aggregated  into  a  single  met¬ 
ric  that  quantifies  the  total  performance  of  each  alternative.  The  composite 
index  is  simply  the  performance  measured  from  the  reference  standard.  We 
call  this  metric  a  performance  index  (PI). 

By  creating  a  single  measure  of  system  performance  rather  than  inde¬ 
pendently  evaluating  multiple  systems’  Performance  Metrics,  we  directly 
compared  different  material  solutions  against  each  other  and  our  reference 
standard  simultaneously.  We  believe  this  analysis  can  assist  other  profes¬ 
sional  decision  analysts,  both  in  military  and  civilian  fields,  to  improve 
the  quality  of  actionable  information  provided  to  leadership.  Further,  the 
graphical  displays  we  created  easily  communicate  complex  economic  trade¬ 
offs  among  capital  equipment  options.  To  illustrate  our  method,  we  apply  it 
to  a  case  study  on  Marine  Corps  vehicles. 
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Case  Study 

Background 

We  began  this  analysis  while  conducting  financial  analytics  and 
modeling  for  the  2014  Marine  Corps  GCT  V  strategy  update  (a  25-year  cap¬ 
ital  investment  plan).  The  Marine  Corps  was  considering  two  investment 
courses  of  action  (COA)  to  recapitalize  its  truck  fleet  with  a  mix  of  newly 
procured  vehicles  and  sustainment  programs  for  older  platforms. 

We  will  call  the  trucks  in  this  fleet  ALPHAs,  BRAVOs,  and  CHARLIEs.  The 
baseline  COA  consisted  of  a  three-platform  mix  of  approximately  20,000 
vehicles.  One-third  would  be  next-generation  BRAVOs;  the  next  third 
would  be  an  upgrade  of  newer  existing  ALPHAs;  and  the  final  third  would 
be  CHARLIEs  undergoing  minimal  sustainment  actions.  This  mixed  fleet 
was  institutionally  preferred  because  it  was  believed  to  provide  acceptable 
performance  at  lower  cost  due  to  the  smaller  quantity  of  BRAVOs  (the  most 
expensive  vehicle).  The  Marine  Corps  was  also  considering  a  second  COA 
that  would  eventually  replace  all  ALPHAs  and  CHARLIEs  with  BRAVOs  by 
2040.  This  COA  would  initially  fund  ALPHAs  and  CHARLIEs,  but  eventu¬ 
ally  replace  them,  one  for  one,  with  BRAVOs. 

By  using  our  PI  methodology,  we  found  that  the  BRAVO  significantly  out¬ 
performs  the  ALPHA  and  CHARLIE  in  an  absolute  sense  and  additionally 
provides  greater  performance  per  dollar  (PPJ  when  its  PI  is  divided  by  the 
procurement  cost.  Additionally  we  wanted  to  explore  whether  other  vehicle 
mixes  could  provide  higher  levels  of  truck  fleet  performance  at  lower  cost, 
because  funding  constraints  often  prohibit  purchasing  desired  quantities 
of  exquisite  systems.  We  used  the  PI  to  generate  four  additional  COAs 
that  showed  higher  return  is  achievable  for  less  cost.  The  Marine  Corps 
subsequently  changed  its  truck  procurement  strategy  partly  because  of 
our  analysis. 

Assumptions 

To  facilitate  our  analysis,  we  made  the  following  assumptions: 

•  BRAVO’s  performance  represented  the  ideal  performance 
benchmarks,  because  without  financial  constraints,  the  only 
vehicle  acquired  would  be  BR AV O. 

•  Performance  Metric  values  had  linear  returns  to  scale. 

•  Performance  Metrics  are  independent,  allowing  them  to  be 
summed  together  in  a  linear  fashion. 


Defense ARJ,  July  2015,  Vol.  S3No.  3:386-349 


July  2015 


•  All  Performance  Metrics  comprised  equal  value,  or  weight, 
relative  to  the  vehicles’  total  performance. 

Research  Questions 

By  generating  a  single  ROI  for  each  platform,  we  could  do  the  following: 

•  RQ1:  Compare  platforms  as  complete  systems,  not  just  indi¬ 
vidual  characteristics  between  systems  (X  vs.  Y,  not  just  the 
payload  of  X  vs.  the  payload  of  Y). 

•  RQ2:  Determine  the  average  PP  spent  for  each  vehicle  alterna¬ 
tive  and  each  vehicle  mix  COA. 

•  RQ3:  Compared  with  the  baseline  COAs,  create  new  ways  to 
achieve  equal  or  greater  performance  in  the  truck  fleet  at  dif¬ 
ferent  funding  levels  as  well  as  identify,  quantify,  and  evaluate 
the  risks  in  all  COAs. 

Methodology 

Data,  variables,  and  modeling.  We  gathered  institutionally  approved 
life-cycle  costs  and  performance  data  for  all  trucks.  We  conducted  all  our 
analysis,  modeling,  and  additional  COA  development  in  Microsoft  Excel. 

Defining  Performance  Metrics.  Table  1  shows  the  six  notional  primary 
Performance  Metrics  for  all  three  trucks  (T).  AR1  and  AR2  are  quantitative 
measures  of  vehicle  armor.  Payload  (D)  is  the  vehicle’s  useful  carrying 
capacity  measured  in  pounds.  Mobility  (M)  is  an  index  value  measuring  the 
vehicle’s  ability  to  maneuver  in  soft  soil.  Reliability  (R)  is  the  mean  miles 
between  operational  mission  failures.  Power  generation  (G)  is  the  number 
of  gallons  per  hour  required  to  produce  20  kilowatts  of  electricity.  For  both 
M  and  G,  a  lower  number  is  better.  Because  each  Performance  Metric  is 
expressed  in  different  units  of  measure,  they  cannot  be  combined  into  an 
aggregate  score  without  normalization. 


TABLE  1. 

VEHICLE  PLATFORMS,  PERFORMANCE  METRICS,  AND  VARIABLES 

Armor 

1  (AR1) 

Armor 

2  (AR2) 

Payload 

(D) 

Mobility 

(M) 

Reliability 

(R) 

Power 

Generation 

(G) 

Total 

Platform 

Performance 

Optimal 

Performance 

0.5 

1.0 

3500.0 

25.0 

2400.0 

1.6 

N/A 

Alpha  (A) 

0.0 

0.4 

1500.0 

30.0 

1100.0 

3.0 

N/A 

Bravo  (B) 

0.5 

1.0 

3500.0 

25.0 

2400.0 

1.6 

N/A 

Charlie  (C) 

0.0 

0.4 

-1155.0 

36.0 

170.0 

3.0 

N/A 
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Normalizing  Performance  Metrics.  Because  BRAVO's  performance 
was  the  standard,  we  turned  its  Performance  Metrics  into  the  variable 
optimal,  0.,  from  which  all  deviations  are  assessed.  Other  applications  of  this 
methodology  could  choose  a  different  0.,  such  as  acquisition  threshold  or  the 
objective  requirements  for  KPPs  and  KSAs.  All  three  trucks'  Performance 
Metrics  are  indexed  against  all  six  optimal  Performance  Metrics  as  a  per¬ 
centage  deviation  in  actual  performance. 


The  general  equation  for  indexing  each  Performance  Metric  (PM)  when  a 
higher  value  is  preferred  was: 


Indexed  PM  = 


Measured  PM 
Optimal  PM 


X100 


For  example,  the  optimal  payload  is  3,500  lb.  The  ALPHA  payload  is  1,500 
lb.  Therefore, 


ALPHA  Indexed  D=  (1500/3500)  *  100  =  42 

This  shows  that  ALPHA  delivers  42  percent  of  our  optimized  payload  (D). 
For  consistency,  we  needed  to  index  mobility  (M)  and  power  generation  (G) 
differently  because  a  lower  score  for  those  metrics  indicates  better  perfor¬ 
mance.  To  do  this,  we  calculated  the  index  so  that  a  measured  Performance 
Metric’s  percentage  difference  above  the  optimal  value  was  an  index  score 
below  100. 


The  general  equation  for  indexing  each  Performance  Metric  when  a  lower 
value  indicates  abetter  score  is: 


Indexed  PM  =  (  2  -  Meas“refPM  )  x  100 

Optimal  PM 

For  example,  the  optimal  (M)  is  25  and  ALPHA’S  M  is  30.  Therefore, 

ALPHA  Indexed  M=  (2  --§§-)  x  100  =  80 

This  shows  that  ALPHA  delivers  80  percent  of  our  optimal  (M).  Table  2 
shows  the  indexed  performance  characteristics  including  the  total  perfor¬ 
mance  of  each  platform.  The  total  platform  performance  (TPP)  column  is 
the  sum  product  of  all  indexed  Performance  Metrics. 

Scaling  and  weighting  performance  variables.  As  stated  in  Table  2, 
TPP  is  an  absolute  measurement  scale;  as  such,  it  is  difficult  to  interpret  and 
gauge  the  percentage  difference  between  platforms.  Scaling  TPP  to  a  100- 
point  scale  increases  the  metric’s  understandability.  Including  the  weights 
(W)  of  the  indexed  Performance  Metrics  allows  their  relative  importance  to 
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affect  the  TPP  score  according  to  organizational  value.  In  this  instance,  we 
did  not  have  institutional  value  assessments  for  the  Performance  Metrics 
and  thus  we  let  all  variables  carry  an  equal  weight  of  16.6  percent  (100/6). 


TABLE  2.  INDEXED  PERFORMANCE  METRICS  AND  ABSOLUTE  TOTAL 

PERFORMANCE 

Armor 

1  (AR1) 

Armor 

2  (AR2) 

Payload 

(D) 

Mobility 

(M) 

Reliability 

(R) 

Power 

Generation 

(G) 

Total 

Platform 

Performance 

Optimal 

Performance 

100 

100 

100 

100 

100 

100 

600 

Alpha  (A) 

0 

40 

43 

80 

46 

13 

222 

Bravo  (B) 

100 

100 

100 

100 

100 

100 

600 

Charlie  (C) 

0 

40 

-33 

56 

7 

13 

83 

The  scaled  TPP  of  each  vehicle  is  the  sum  product  of  its  indexed  Performance 
Metrics.  Using  the  equal  Performance  Metric  weights  (16.6  percent),  the 
TTP  equation  is: 

TPP  =  Wppu*  Indexed  FPU  +  WFpw*  Indexed  FPW  +  WD* 

Indexed  D  +  W*  Indexed  M  +  Wa  *  Indexed  R  +  W„  *  Indexed  G 

Table  3  shows  the  scaled  and  weighted  TPP  scores,  while  Figures  1  and  2 
depict  the  DTR  method  and  show  how  the  scores  are  plotted.  Figure  2  shows 
the  TPP  scores  for  the  three  alternatives  against  the  optimal.  It  is  important 
to  note  that  the  payload  of  Charlie,  even  with  additional  maintenance  fund¬ 
ing,  returns  less  payload  than  required:  this  Performance  Metric  returns 
negative  value  and  is  not  a  computational  error. 


TABLE  3.  INDEXED  AND  SCALED  PERFORMANCE  METRICS 

Armor 

1  (AR1) 

Armor 

2  (AR2) 

Payload 

(D) 

Mobility 

(M) 

Reliability 

(R) 

Power 

Generation 

(G) 

Total 

Platform 

Performance 

Optimal 

Performance 

17 

17 

17 

17 

17 

17 

100 

Alpha  (A) 

0 

7 

7 

13 

8 

2 

37 

Bravo  (B) 

17 

17 

17 

17 

17 

17 

100 

Charlie  (C) 

0 

7 

-6 

9 

1 

2 

14 
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FIGURE  1.  DISTANCE  TO  REFERENCE  METHOD 


Determining  the  truck  fleet’s  cumulative  performance  return.  After 
deriving  the  TPPs  for  each  platform,  we  calculated  each  COA’s  impact  on 
truck-fleet  performance  by  year,  and  over  the  entire  investment  horizon. 
Mathematically,  this  is  vehicle  quantity  multiplied  by  TPP  Total  cumula¬ 
tive  performance  is  how  much  value  a  COA  generates  within  the  Marine 
Corps  truck  fleet. 
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We  multiplied  the  projected  yearly  fleet  mix  inventory  (I),  which  is  the 
planned  procurement  quantity  plus  existing  inventory,  by  TPP  A  ,,  f  ,to  obtain 
a  yearly  COA  Performance  Point  Score  ( CPPS F)  for  each  year  between  2015 
and  2040.  CPPS  Fis  the  aggregate  return  of  the  entire  truck  fleet  in  one  year. 
We  then  summed  all  the  years  to  obtain  a  total  score  for  each  COA  ( CPPS ^ 
CPPST  represents  how  much  total  value  each  COA  could  provide  across  the 
entire  investment  horizon  (2015-2040).  The  equations  for  CCPSyT  are: 

CPPSy  =  X  tppab  c*  iab  c 

CPPS,  =  X  CPPS 

yiS01S-S040 

Determining  cost.  To  develop  total  cost,  we  multiplied  the  yearly  inven¬ 
tory  (I)  for  all  platforms IA  B  c by the  projected  maintenance  and  procurement 
actions  in  a  given  year.  We  then  calculated  the  yearly  costs  (C^)  and  total 
costs  (C,()  for  each  COA.  Table  4  lists  the  variables,  factors,  and  costs  associ¬ 
ated  with  the  truck  fleet.  All  costs  are  in  thousands  of  calendar  year  2014 
dollars.  The  “New”  column  shows  the  cost  of  procuring  a  single  BRAVO  and/ 
or  ALPHA.  The  columns  for  “SLEP”  (Service  Life  Extension  Program)  show 
the  cost  of  conducting  a  major  overhaul  of  the  CHARLIE  fleet  and  the  per¬ 
centage  ofIA  overhauled  each  year.  The  columns  under  “IROAN”  (Inspect 
or  Repair  Only  as  Needed)  show  the  estimated  cost  and  percentage  of  each 
fleet  scheduled  for  IROAN  maintenance  actions  each  year.3  CN  is  multiplied 
by  the  yearly  acquisition  quantities.  For  brevity,  Table  5  lists  only  the  begin¬ 
ning  and  final  acquisition  quantities. 


TABLE  4.  VARIABLES,  COSTS,  AND  FACTORS 

Buy  New 
(N) 

SLEP  (S) 

IROAN  (IR) 

CY14$K 

Cost  (CN) 

%°f 

°S  Inventory/ 

{  Year  (1,) 

Cost  (CIR) 

%of 

Inventory/ 
Year  (/,„) 

Alpha 

$175 

—  — 

$86 

5% 

Bravo 

$435 

-  — 

$115 

5% 

Charlie 

— 

$147  5% 

$100 

10% 

The  platform  cost  equations  for  BRAVO  and  ALPHA  are: 
BravoC^UC„ *£„„)  + 
AlphaCyUCJAn)HCm’IIR) 
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I  TABLE  5.  TOTAL  COST  FOR  COURSES  OF  ACTION:  2015-2020  J 

COA 

Total  Cost  (FY14$B) 

1 

$7.10 

2 

$11.90 

3 

$6.90 

4 

$8.30 

5 

$4.70 

6 

$10.20 

The  platform  cost  equation  for  CHARLIE,  which  does  not  have  any  new 
procurement,  is: 

Charlie  C  =  I  (Cs  */s)  +  (C„  •/„) 

The  total  cost  (Cr)  for  a  COA  is: 

Cr=  X  Cy 

n-2015-2040  A,  B,  C 


Table  6  shows  the  total  cost  for  each  COA  from  2016  to  2040. 


TABLE  6.  TRUCK  FLEET  INVESTMENT  COAS 

COA 

COA  Description 

Alpha  QTY 

Bravo  QTY 

Charlie  QTY 

Total 

o 

2016 

2040 

2016 

2040 

2016 

2040 

2040 

c 

a> 

</) 

fU 

1 

Mixed  Fleet 

1325 

6851 

650 

5500 

16350 

5421 

17772 

Cfl 

2 

Max  Alpha/Max  Bravo 

1325 

0 

97 

17772 

16350 

1972 

17772 

2 

No  Bravo  (Null) 

1325 

6851 

0 

0 

16447 

10921 

17772 

■o 

o 

** 

ru 

<D 

4 

Cancel  Alpha,  Divert 

Funds  to  Bravo 

0 

0 

650 

7121 

17122 

10651 

20242 

u 

111 

<* 

< 

5 

Cancel  Alpha  &  Charlie, 
Buy  More  Bravo 

0 

0 

650 

7121 

17122 

1473 

17772 

a 

6 

Cancel  Alpha  &  Charlie, 
Buy  Full  Bravo 

0 

0 

2173 

17772 

18468 

1473 

17772 
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Calculating  PP$.  The  CT  of  a  COA  is  the  sum  of  all  procurement  and  depot- 
level  sustainment  actions  from  2015  to  2040.  To  create  a  normalized  return 
per  dollar  spent  for  each  COA,  we  divided  CCPST  by  Cr  We  call  this  metric 
performance  points  per  dollar,  PP  This  provides  a  normalized  measure  of 
the  performance  return  for  each  dollar  spent  on  a  truck  and  the  entire  fleet 
capability.  A  higher  value  is  better  than  a  lower  one  because  it  indicates 
greater  performance  for  less  money. 


The  PP$  equation  for  a  COA  and  individual  truck  is: 

COA  Return  =  PP=  CPPSt 
1-6 

C  T 
1-6 


Truck  Return  =  PP$T= 


_ TPP _ 

Average  Procurement  Unit  Cost  ( APUC ) 


Analysis 

Truck  Investment  COAs 

We  began  the  analysis  with  only  two  COAs  in  the  truck  portfolio. 
However,  we  created  four  additional  COAs  to  evaluate  cost  and  performance 
changes  by  varying  fleet-mix  (see  Table  6).  Specifically,  we  constructed  a 
null  COA  to  assess  the  truck  fleet’s  value  without  BRAVOs  and  three  addi¬ 
tional  COAs  to  explore  the  potential  trade-offs  amongthe  other  alternatives. 
The  primary  factor  driving  new  COA  development  was  leadership’s  desire  to 
understand  how  many  more  BRAVOs  could  be  bought  if  funding  planned  for 
ALPHAS  and  CHARLIEs  was  redirected  to  BRAV O  procurement  instead. 

Findings 

By  using  our  method,  we  evaluated  our  three  research  questions. 

For  RQ1,  we  found  that  the  TPPs  are  as  follows: 

1.  TPPb  =  100 

2.  TPPa  =  37 

3.  TPPC  =  14 

ALPHA  is  only  one-third  as  capable  as  BRAVO.  CHARLIE  is  only  one- 
seventh  as  capable. 

For  RQ2,  by  dividing  TPPs  by  APUC  (Average  Procurement  Unit  Cost)4and 
SLEP  costs  listed  in  Table  4,  we  calculated  their  performance  points  per 
(thousand)  dollars: 
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BRAVO  PP  =1.38 
ALPHA  PP  =1.27 

>p 

CHARLIE  PP 3=0.56 

These  metrics  imply  that  every  thousand  dollars  spent  on  the  BRAVO 
returns  approximately  1.38  performance  points;  on  the  ALPHA,  1.27;  and 
on  the  CHARLIE,  0.56.  In  this  example,  spending  money  on  ALPHAS  or 
CHARLIEs  does  not  offer  the  highest  return.  Funding  should  thus  be  spent 
on  BRAVOs  instead.  However,  this  analysis  also  showed  that  even  though 
overall  ALPHA  performance  is  less  than  that  of  BRAVOs,  its  low  APUCa 
relative  to  APUCB,  allows  for  the  creation  of  a  COA  with  a  similar  level  of 
PP$  at  less  cost. 

We  also  compared  the  sensitivity  of  PP$  to  changes  in  the  APUC.  We  found 
that  when  the  APUC  for  ALPHA  falls  below  $160,200,  a  BRAVO  no  longer 
offers  the  highest  return  per  dollar.  Figure  3  shows  the  performance  and 
cost  curves  for  BRAVO  and  ALPHA.  These  curves  identify  the  change  in 
PP$  relative  to  the  APUC  as  well  as  the  inflection  points  where  changes  in 
the  APUC  reverse  our  previously  stated  best-value  assessment.  The  Y-axis 
is  TPP  per  $1,000  and  the  X-axis  is  a  range  of  APUCs.  BRAVO’s  performance 
is  superior  in  absolute  terms,  but  PP$  is  sensitive  to  differences  in  unit  cost. 
The  current  unit  cost  estimates  for  ALPHAS  and  BRAVOs  are  roughly  pro¬ 
portional  to  their  relative  absolute  performance  levels,  explaining  why  the 
PP$  is  close  between  the  options.  However,  changes  in  those  unit  cost  esti¬ 
mates  would  change  the  PP,  even  if  absolute  performance  does  not  change. 

Ip 


FIGURE  3.  PERFORMANCE  AND  COST  CURVES  FOR  BRAVO  AND 

ALPHA 


APUCs (FY14$  Thousands) 

—  BRAVO  Performance  Score  /  $1000  —  ALPHA  Performance  Score  /  $1000 
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Holding  all  other  variables  constant,  we  can  also  see  that  the  BRAVO  would 
not  offer  highest  PP$  if  APUCB  exceeds  $474,400.  A  rise  of  $39,400,  or  9 
percent,  in  APUCB,  makes  the  ALPHA  abetter  alternative  per  dollar.  The 
change  in  PP$  due  to  change  in  APUCB ,  is  shown  below: 

BRAVO  PP$=  1.26 

ALPHA  PP  =1.27 

>p 

Conversely,  if APUCB  is  held  constant  at  $435,000,  the  APUC  for  the  ALPHA 
must  drop  by  approximately  8  percent  to  $160,200  for  it  to  become  the  better 
option  in  terms  of  PP$. 

Finally,  for  RQ3,  we  evaluated  all  COAs’  performance  in  four  ways:  (a) 
performance  return  each  year  from  2015  to  2040,  CPPSy;  (b)  performance 
levels  of  each  COA,  CPPST;  (c)  cost  of  each  COA,  CT;  and  (d)  average  trade-off 
between  COAs,  PP,  (see  Table  7).  This  absolute  scale  shows  the  magnitudes 
of  differences  between  the  COAs  in  terms  of  cost  and  performance  return. 
For  example,  we  can  see  that  COA6  provides  twice  the  total  performance 
of  COA3  for  an  additional  $3.3  billion. 

Figure  4  illustrates  the  information  presented  in  Table  7.  The  X  axis  shows 
the  total  performance  points  over  the  investment  period;  the  Y  axis  shows 
the  total  cost  of  each  COA  in  billions  of  constant  year  2014  dollars  from  2015 
to  2040;  and  the  size  of  each  bubble  indicates  the  PP$. 


FIGURE  4.  TRUCK  FLEET'S  COA  PERFORMANCE,  COST,  AND 
PERFORMANCE  PER  DOLLAR 
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|  TABLE  7.  TRUCK  FLEET  COAS'  COST  AND  PERFORMANCE  ] 

COA 

Total  Cost  (CT) 
(FY14SB) 

Cumulative 
Performance  Points 
Total  (CCPST) 

Performance  Points 
per  Dollar  (PP$) 

1 

$7.10 

20,069,918 

2.83 

2 

$11.90 

25,386,519 

2.14 

3 

$6.90 

10,260,405 

1.48 

4 

$8.30 

22,130,037 

2.68 

5 

$4.70 

24,925,012 

5.33 

6 

$10.20 

35,637,929 

3.50 

Using  this  figure  to  evaluate  the  COAs  in  our  example  quickly  leads  to  the 
elimination  of  C0A1-4,  because  COA5  and  6  have  either  greater  capability 
or  lower  cost  and  greater  capability.  This  presents  leadership  with  a  choice 
of  maximizing  performance  overall  (COA6)  or  maximizing  PP$  (COA5).  An 
efficient  leadership  decision  is  thus  between  the  value  of  the  extra  capability 
and  the  opportunity  cost  of  attaining  it. 

Figure  5  shows  the  change  in  average  PP$  incurred  by  moving  among  alterna¬ 
tives.  For  instance,  moving  a  dollar  from  C0A4  to  COA2  buys  0.5  fewer 
performance  points  than  keeping  that  dollar  in  C0A4.  Conversely,  moving  a 
dollar  from  COA2  to  C0A4  buys  0.5  more  performance  points.  Overall,  COA5 
is  the  best,  as  moving  from  COA5  to  any  alternative  reduces  the  average  PP$. 
In  addition,  moving  from  COA6  to  any  other  COA  except  COA5  reduces  the 
net  benefit.  Hence,  COA5  and  6  should  be  the  focus  of  leadership’s  decision 
making,  and  the  original  COAs  (COA1  and  2)  should  be  abandoned. 
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The  timing  of  major  investment  decisions  is  another  important  factor  to 
consider.  Our  methodology  can  be  used  to  evaluate  performance  over  time. 
Figure  6  plots  the  CPPSy  of  COA1-6  from  2016  to  2040.  This  graphical 
representation  shows  each  COA’s  benefit  stream  per  year,  throughout  the 
investment  horizon.  Several  options  provide  more  performance  earlier,  and 
at  less  cost  than  the  current  baseline  plan. 


FIGURE  6.  TRUCK  FLEET  COAS'  PERFORMANCE  BY  YEAR 


COA  3  $8.08  C0A4  $10.08 

2  Extend  BRAVO  procurement 

-  —  COA 5  $5.68  ■  —  —COA 6  $10.08 


Decision  point  1  shows  where  leadership  should  abandon  C0A1-4.  Decision 
point  2  shows  when  a  choice  between  COA5  and  6  should  be  made.  This 
graphic  highlights  when  major  decisions  are  required  before  reductions 
in  capability  may  appear  and  focuses  discussions  on  risks,  trade-offs,  and 
mitigation  plans  across  the  time  horizon.  Leaders  can  assess  differences  in 
each  COA’s  performance  by  year,  over  time,  and  against  total  cost.  Aggregate 
cost  and  fleet  performance  trade-offs  can  be  evaluated  simultaneously. 
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Additional  Applications  of  the 
Methodology 

Source  Selection  Decisions 

Using  the  PI  method  during  formal  source-selection  decisions  could 
allow  all  potential  platforms  under  consideration  to  be  normalized  and 
evaluated  objectively  with  (a)  a  common  performance  scale,  and  (b)  a  single 
metric  based  on  each  platform’s  ROI  relative  to  performance  standards.  PP„ 
shows  each  system’s  monetized  performance  return  and  stream  of  benefits 
over  time,  allowing  for  direct  comparisons  against  all  competitors.  In  addi¬ 
tion,  as  we  have  shown,  this  methodology  defines  the  APUC  range  within 
which  a  given  system  is  the  preferred  option  in  terms  of  PP$.  The  PI  method 
can  also  facilitate  objective  strategic  discussions  about  how  different  sys¬ 
tems  affect  the  projected  fleet’s  performance. 

This  methodology  is  not  solely  limited  to  the  DoD  or  military  acquisition 
process.  Any  entity  that  makes  capital  investments  that  do  not  produce 
a  monetized  return  could  use  it  to  compare  alternatives  objectively.  For 
example,  municipal  governments  and  public  safety  agencies  procuring 
emergency  equipment  could  benefit  from  using  this  methodology,  especially 
if  there  is  not  a  formally  defined  or  rigorous  acquisition  process  and  lead¬ 
ers  simply  want  to  know  if  they  are  getting  “the  most  bang  for  the  buck.” 
The  basic  requirement  is  to  understand  the  desired  goals,  objectives,  and 
performance.  If  an  investment  has  required  performance  standards,  each 
alternative’s  deviation  from  that  standard  is  straightforward  and  easily 
calculated  using  common  software. 


Limitations  of  the  Methodology 

As  presented,  the  method  has  several  areas  for  improvement.  First,  the 
assumption  of  linear  returns  to  scale  possibly  overstates  the  scores  for  each 
platform.  However,  this  effect  can  be  mitigated  by  including  weights  that 
reflect  institutional  value  functions.  MODA  and  AHP  are  effective  methods 
to  develop  institutional  value  functions  on  each  Performance  Metric  for 
inclusion  in  the  PI  calculation.  In  addition,  the  weights  of  each  Performance 
Metric  should  accurately  reflect  their  value  contribution  to  the  system’s 
total  performance. 
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Conclusions 

We  used  disparate  performance  measures  to  calculate  a  composite 
PI— an  ROI  proxy—  to  analyze  the  nonmonetized  return  of  three  trucks. 
We  then  evaluated  two  institutionally  directed  COAs  for  truck  procure¬ 
ment  and  developed  additional  COAs  with  different  cost  and  performance 
trade-offs.  We  found  that  the  existing  COAs  provide  less  performance  for 
more  cost  than  the  alternatives.  A  COA  we  created  represented  the  most 
efficient  use  of  fiscal  resources  since  it  provided  the  second  highest  level  of 
performance  at  almost  half  the  cost  of  the  other  COAs.  This  analysis  shows 
that  the  Marine  Corps  can  return  more  performance  for  each  dollar  spent. 
We  recommended  that  the  Marine  Corps  reevaluate  its  truck  fleet  options 
and  consider  alternative  COAs.  Based  in  part  on  this  analysis,  the  Marine 
Corps  shifted  procurement  plans  in  the  GCTV  fleet. 


This  PI  method  is  objective:  it  is  simply  a  reflection 
of  institutional  requirements  and  tested  system 
performance. 


This  method  also  has  broad  analytical  applicability.  First,  the  power  of  this 
method  lies  in  its  ability  to  aggregate  disparate  performance  measurements 
into  a  common  scale.  This  PI  method  is  objective:  it  is  simply  a  reflection  of 
institutional  requirements  and  tested  system  performance.  By  removing 
subjective  bias,  equipment  investment  decisions  can  focus  on  salient  issues 
(e.g.,  cost,  performance)  rather  than  the  different  value  perceptions  among 
stakeholders.  Second,  as  an  ROI  metric,  the  PI  highlights  areas  of  oppor¬ 
tunity  and  loss.  Options  that  inefficiently  spend  funding  can  be  eliminated 
early,  allowing  subsequent  analytical  efforts  to  focus  on  alternatives  return¬ 
ing  highest  institutional  value.  This  improves  decision  quality  and  speed. 
Finally,  using  the  PI  method  and  graphics  in  this  article,  complex  economic, 
cost,  and  performance  information  can  be  modeled  quickly,  supporting 
changing  strategies.  Altering  variables  (e.g.,  cost,  vehicle  mix,  time)  allows 
leaders  to  see  the  impacts  their  ideas  have  on  fleet  cost  and  performance, 
and  assess  the  associated  risks.  All  these  factors  lead  to  the  PI  method  as  an 
effective  way  to  determine  the  ROI  of  nonmonetized  investments. 
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Endnotes 

1  "Key  Performance  Parameters  (KPP)  are  those  attributes  or  characteristics  of  a 
system  that  are  considered  critical  or  essential  to  the  development  of  an  effective 
military  capability  and  that  make  a  significant  contribution  to  the  characteristics 
of  the  future  joint  force.  A  KPP  normally  has  a  threshold  representing  the  minimum 
acceptable  value  achievable  at  low-to-moderate  risk  and  an  objective  representing 
the  desired  operational  goal,  but  at  higher  risk  in  cost,  schedule,  and  performance” 
(Hagan,  2009,  p.  B-100). 

2“Key  System  Attributes  (KSA)  are  the  attributes  considered  most  critical  or 
essential  for  an  effective  military  capability,  but  not  selected  as  Key  Performance 
Parameters  (KPP)”  (Hagan,  2009,  p.  B-101). 

3"A  service  life  extension  program  or  SLEP  is  a  major  overhaul  of  a  vehicle  that 
incorporates  reengineering,  modification  and  other  activities  with  the  goal  of 
extending  the  useful  life  of  the  vehicle.  Alternatively,  an  Inspect  or  Repair  Only  as 
Needed,  or  IROAN,  is  a  much  more  limited  program  that  only  replaces  components 
as  required  and  does  not  feature  reengineering  or  modification”  (Hagan,  2009). 

4“Average  Procurement  Unit  Cost  (APUC)  is  calculated  by  dividing  total 
procurement  cost  by  the  number  of  articles  to  be  procured”  (Hagan,  2009,  p. 

B-15). 
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Review: 

Richard  Whittle’s  The  Dream  Machine  is  as  close  to  a  comprehensive 
review  of  a  defense  acquisition  program  as  we  are  likely  to  find  in  our  cur¬ 
rent  “sound  byte”-focused  culture.  It  traces  the  controversial  and  frequently 
maligned  V-22  Osprey  program  from  its  earliest  days  to  its  vindication  in 
2011  after  successful  deployments  to  Iraq  and  Afghanistan.  Whittle  does 
this  through  the  eyes  of  the  key  personalities  in  industry,  government,  and 
the  U.S.  Marine  Corps  who  made  the  “dream”  of  tilt  rotor  technology  into  a 
reality  with  some  of  them  giving  their  lives  in  the  process. 

One  of  the  key  figures  profiled  in  the  book  is  Richard  “Dick”  Spivey  who  started 
in  1959  as  an  18-year-old  Georgia  Tech  “co-op”  student  at  Bell  Helicopter  in 
Fort  Worth,  Texas,  worked  his  way  up  to  “sales  engineer”  for  the  newtilt  rotor, 
started  a  family,  divorced,  remarried,  retired,  came  back  as  a  consultant,  and 
retired  for  good  in  2006— all  before  the  V-22  achieved  its  initial  operational 
capability.  During  his  tenure  at  Bell  Helicopter,  he  traveled  all  over  the  world 
giving  over  2,000  tilt  rotor  briefings  and  sales  presentations. 

The  book  is  organized  into  12  chapters,  each  one  covering  a  specific  facet 
of  the  V-22  story.  For  example,  Chapter  One  “The  Dream”  traces  the  early 
attempts  to  develop  a  “convertiplane,”  which  although  unsuccessful,  still 
offered  the  promise  that  such  technologies  could  eventually  be  made  to 
work.  Chapter  Two  “The  Salesman”  uses  Dick  Spivey’s  career  to  illustrate 
the  opportunistic,  but  persistent  process  used  by  defense  contractors  to 
market  their  products  to  the  military.  Chapter  Three  “The  Customer”  traces 
the  convoluted  requirements  development  and  procurement  processes  used 
by  the  militaryto  acquire  the  equipment  they  think  they  need.  Chapter  Four 
“The  Sale”  shows  the  extremely  lengthy,  but  clever  approach  used  by  Bell 
and  Boeing  to  market  their  immature  tilt  rotor  technology  into  a  systems 
contract  with  the  Marine  Corps.  Other  chapters  detail  the  engineering  trade¬ 
offs  and  political  compromises  made  during  development  of  the  first  V-22 
prototypes.  The  author  also  provides  detailed  accounts  of  the  major  aircraft 
crashes  that  occurred  as  a  result  of  tight  funding,  design  compromises,  and 
accelerated  development,  which  drove  the  program  in  its  early  years. 

The  author’s  ability  to  integrate  the  key  personalities  that  shaped  the  V-22 
program  into  a  chronological  narrative  that  includes  all  major  program 
events  makes  this  not  just  a  historical  account,  but  a  fascinating  and  insight¬ 
ful  look  at  how  dysfunctional  our  “military-industrial”  complex  has  become. 
But  with  further  reflection  and  analysis,  the  story  of  The  Dream  Machine 
can  also  help  us  find  the  way  forward  to  construct  an  improved  acquisition 
process,  which  will  help  us  deliver  our  future  dream  machines. 
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